Oct 30 08:53:13.985969 ip-10-0-129-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:53:14.407191 ip-10-0-129-105 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:14.407191 ip-10-0-129-105 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:53:14.407191 ip-10-0-129-105 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:14.407191 ip-10-0-129-105 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:14.412493 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412403 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.412493 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412493 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.412493 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412496 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412500 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412504 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412506 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412509 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412512 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412515 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412519 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412524 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412528 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412531 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412534 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412537 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412540 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412542 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412545 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412553 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412557 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412559 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412562 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.412590 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412565 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412568 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412570 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412573 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412575 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412578 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412581 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412583 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412586 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412588 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412591 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412594 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412598 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412600 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412603 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412606 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412609 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412611 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412614 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.413076 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412617 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412619 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412622 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412624 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412627 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412629 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412632 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412634 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412637 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412640 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412643 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412645 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412648 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412653 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412656 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412660 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412663 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412665 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412668 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.413533 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412670 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412685 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412688 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412691 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412694 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412696 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412699 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412702 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412704 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412708 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412711 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412714 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412716 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412719 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412722 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412724 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412729 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412732 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412735 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.414008 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412737 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412740 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412743 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412747 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412749 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412752 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412755 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412758 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412761 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412764 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412766 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412769 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412772 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412774 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412777 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412782 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412785 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412787 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412790 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.414517 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412793 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412795 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412798 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412802 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.412804 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413587 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413593 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413597 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413600 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413602 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413605 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413608 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413611 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413613 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413616 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413619 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413622 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413625 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413628 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.414997 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413632 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413634 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413637 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413641 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413643 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413646 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413649 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413654 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413657 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413660 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413662 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413665 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413668 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413670 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413687 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413690 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413693 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413696 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.415461 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413698 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413701 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413703 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413706 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413709 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413713 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413718 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413721 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413724 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413726 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413730 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413732 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413736 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413739 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413741 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413744 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413747 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413749 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413752 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.415927 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413754 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413757 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413759 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413762 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413765 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413768 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413770 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413773 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413775 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413777 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413780 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413782 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413785 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413788 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413790 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413793 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413795 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413798 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413800 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413803 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.416375 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413806 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413808 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413811 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413814 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413817 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413822 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413825 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413828 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413830 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413833 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413837 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413839 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413842 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413844 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413847 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413850 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413852 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413856 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.416867 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413860 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413863 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413866 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413869 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413871 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413874 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413876 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413879 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413881 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413884 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413887 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413889 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413892 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413896 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.413899 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414481 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414489 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414498 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414501 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414505 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.417324 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414508 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414511 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414513 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414516 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414519 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414522 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414524 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414527 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414530 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414533 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414536 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414538 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414541 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414544 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414546 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414549 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414551 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414554 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414556 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414559 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.417803 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414561 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414564 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414566 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414569 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414572 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414575 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414578 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414580 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414583 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414585 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414593 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414596 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414599 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414602 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414604 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414607 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414610 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414612 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414615 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.418261 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414618 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414622 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414631 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414634 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414636 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414639 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414641 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414644 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414647 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414651 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414655 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414657 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414660 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414663 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414665 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414668 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414671 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414687 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414690 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.418724 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414692 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414695 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414697 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414700 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414702 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414711 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414714 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414717 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414719 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414722 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414724 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414727 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414729 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414732 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414734 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414737 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414739 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414742 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414744 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.419277 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414747 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414753 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414755 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414758 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414760 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414763 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414765 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414771 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414773 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414777 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414780 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414783 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414785 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414788 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414791 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414796 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414799 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414807 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.419750 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414811 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414814 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414816 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.414819 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415956 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415970 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415984 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415988 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415993 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.415997 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416002 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416006 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416010 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416013 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416016 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416019 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416023 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416026 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416029 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416032 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416036 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416039 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416047 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416050 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:53:14.420218 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416053 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416056 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416059 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416064 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416067 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416070 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416073 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416076 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416085 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416088 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416091 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416094 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416098 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416101 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416105 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416107 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416111 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416113 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416122 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416125 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416128 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416131 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416134 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416138 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:53:14.420801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416141 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416144 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416147 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416150 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416154 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416157 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416160 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416163 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416166 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416168 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416173 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416176 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416179 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416182 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416185 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416188 2577 flags.go:64] FLAG: --help="false" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416191 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416200 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416204 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416207 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416210 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416213 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416217 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:53:14.421389 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416219 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416222 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416226 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416229 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416232 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416235 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416238 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416241 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416244 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416247 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416250 2577 flags.go:64] FLAG: --lock-file="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416253 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416256 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416259 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416265 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416268 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416271 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416274 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416277 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416281 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416283 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416286 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416291 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416294 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416298 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 08:53:14.421944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416301 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416304 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416312 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416316 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416318 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416321 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416324 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416332 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416335 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416338 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416341 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416344 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416347 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416349 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416353 2577 flags.go:64] FLAG: --port="10250" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416356 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416359 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7ca2e4dd64f4aae" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416362 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416365 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416368 2577 flags.go:64] FLAG: --register-node="true" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416371 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416375 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416378 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416381 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416384 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416388 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:53:14.422581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416391 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416394 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416397 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416400 2577 flags.go:64] FLAG: --runonce="false" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416403 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416406 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416409 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416412 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416415 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416424 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416427 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416431 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416434 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416436 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416440 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416442 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416446 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416449 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416451 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416456 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416459 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416462 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416469 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416471 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416474 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:53:14.423343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416477 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416482 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416486 2577 flags.go:64] FLAG: --v="2" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416491 2577 flags.go:64] FLAG: --version="false" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416495 2577 flags.go:64] FLAG: --vmodule="" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416499 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.416502 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416723 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416728 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416732 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416735 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416737 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416740 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416743 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416746 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416749 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416751 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416760 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416763 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416765 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.424378 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416768 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416772 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416777 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416779 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416785 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416788 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416790 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416793 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416796 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416798 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416801 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416803 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416807 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416811 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416814 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416817 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416819 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.425220 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416827 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416830 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416832 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416835 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416837 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416840 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416842 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416845 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416848 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416850 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416859 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416862 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416865 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416868 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416871 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416874 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416876 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416879 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416882 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.425952 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416884 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416887 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416890 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416892 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416895 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416898 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416901 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416908 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416911 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416914 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416916 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416919 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416922 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416924 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416927 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416930 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416932 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416935 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416938 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.426432 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416941 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416943 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416946 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416948 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416951 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416960 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416963 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416966 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416968 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416971 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416974 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416976 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416979 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416982 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416984 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416986 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416989 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416992 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416994 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.416998 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.426934 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417002 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417005 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417007 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417010 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417013 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417021 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417024 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417026 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417029 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417031 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417034 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417036 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417039 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417042 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417044 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.417070 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:14.427408 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417133 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417138 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417147 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417151 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417155 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417158 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417160 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417163 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417166 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417168 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417171 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417173 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417176 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417179 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417182 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417186 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417188 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417192 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417195 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.427805 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417197 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417200 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417203 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417208 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417211 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417214 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417216 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417219 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417222 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417224 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417227 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417229 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417232 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417235 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417237 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417245 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417248 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417251 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417253 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.428254 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417256 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417259 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417261 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417264 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417266 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417269 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417271 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417274 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417278 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417280 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417285 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417288 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417290 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417293 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417296 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417301 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417303 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417306 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417309 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.428744 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417312 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417315 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417317 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417320 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417322 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417326 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417330 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417333 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417336 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417346 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417349 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417352 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417354 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417357 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417359 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417362 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417365 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417367 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417370 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.429260 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417373 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417376 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417379 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417382 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417385 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417388 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417390 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417393 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417396 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417398 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417401 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417404 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417406 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417409 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417411 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417414 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417417 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417420 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417423 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417426 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.429785 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417428 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417431 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417433 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417442 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417444 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417447 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417449 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.417452 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.426529 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.426547 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.426571 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.426577 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426645 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426656 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426660 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426663 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426665 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426668 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.430252 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426671 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426688 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426691 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426694 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426696 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426699 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426702 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426705 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426708 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426711 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426714 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426717 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426720 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426722 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426725 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426728 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426731 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426733 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426736 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426738 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.430756 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426741 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426744 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426748 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426751 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426753 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426756 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426759 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426762 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426765 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426767 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426770 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426773 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426775 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426778 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426780 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426783 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426786 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426788 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426791 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426793 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.431219 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426796 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426799 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426801 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426804 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426807 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426809 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426812 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426814 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426817 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426819 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426822 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426825 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426828 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426830 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426834 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426837 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426839 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426842 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426845 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426847 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.431706 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426850 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426852 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426855 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426858 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426860 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426863 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426865 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426868 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426871 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426873 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426877 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426881 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426885 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426888 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426891 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426896 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426898 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426901 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.432177 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426903 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426906 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426908 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426911 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426913 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426916 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426918 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426921 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426924 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426927 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426929 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426933 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426935 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426938 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426940 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426945 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426949 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426953 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426956 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.432623 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.426959 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.426987 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427166 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427171 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427175 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427178 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427181 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427184 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427188 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427190 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427193 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427195 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427198 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427201 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427203 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427206 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427209 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.433081 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427212 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427214 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427217 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427220 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427223 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427225 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427228 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427231 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427233 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427236 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427239 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427241 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427244 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427247 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427250 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427252 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427255 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427257 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427260 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427262 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.433487 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427265 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427267 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427270 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427274 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427277 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427279 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427282 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427285 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427287 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427290 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427293 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427296 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427301 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427303 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427306 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427309 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427311 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427314 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.434094 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427317 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427319 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427322 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427324 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427327 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427330 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427332 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427335 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427337 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427340 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427342 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427345 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427347 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427350 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427352 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427355 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427357 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427361 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427364 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427366 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.434537 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427369 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427371 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427376 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427379 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427382 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427384 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427387 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427390 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427392 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427395 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427398 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427400 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427403 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427405 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427408 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427410 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427413 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427416 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427418 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.435024 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427421 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427424 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427426 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427428 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427431 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427433 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427436 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427439 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427442 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427446 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427449 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427453 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427455 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.427478 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427511 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:14.435470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427515 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427518 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427521 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427524 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427526 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427529 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427532 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427534 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427537 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427540 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427543 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427546 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427548 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427551 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427554 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427556 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427559 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427561 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427564 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427566 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:14.435870 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427569 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427572 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427574 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427577 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427580 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427582 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427584 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427587 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427592 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427595 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427598 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427600 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427605 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427608 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427610 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427613 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427615 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427618 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427621 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427624 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:14.436348 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427626 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427629 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427632 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427635 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427637 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427640 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427642 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427645 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427648 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427650 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427653 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427655 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427658 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427661 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427664 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427666 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427669 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427671 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427688 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:14.436829 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427692 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427696 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427699 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427705 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427708 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427713 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427716 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427718 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427721 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427724 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427726 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427729 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427731 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427734 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427737 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427739 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427743 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427746 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427750 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:14.437311 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427752 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427755 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427758 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427761 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427763 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427766 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427768 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427771 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427773 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427776 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427779 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427784 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427787 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427791 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427793 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427796 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427799 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427805 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:14.437850 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427807 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427810 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427813 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427818 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427822 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427825 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427827 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:14.427830 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.427896 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.430391 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.431525 2577 server.go:1014] "Starting client certificate rotation" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.431619 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.431654 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.433854 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:14.438303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.435022 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:14.444447 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.444433 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:53:14.448294 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.448281 2577 log.go:25] "Validated CRI v1 image API" Oct 30 08:53:14.449202 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.449178 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:53:14.453073 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.453053 2577 fs.go:135] Filesystem UUIDs: map[289d09e1-363d-4a8d-b851-a5d8afd10014:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd7b0745-cba1-4abc-bdc5-0a8f1c0c5b7a:/dev/nvme0n1p4] Oct 30 08:53:14.453155 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.453070 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 08:53:14.460071 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.459952 2577 manager.go:217] Machine: {Timestamp:2026-10-30 08:53:14.458418305 +0000 UTC m=+0.374732634 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099677 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e3bc00e65b2a8a7adee1566c8945 SystemUUID:ec23e3bc-00e6-5b2a-8a7a-dee1566c8945 BootID:35721d21-7570-460b-8399-2673731b2c9b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:c7:82:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:c7:82:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:60:f6:90:1d:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:53:14.460894 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.460883 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:53:14.460996 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.460981 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:53:14.464682 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464645 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:53:14.464861 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464692 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:53:14.464908 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464867 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:53:14.464908 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464875 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:53:14.464908 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464900 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:14.464981 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.464912 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:14.466160 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.466149 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:53:14.470756 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.470743 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:53:14.470914 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.470905 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:53:14.471756 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.471746 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:53:14.471795 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.471768 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:53:14.471795 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.471780 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:53:14.471795 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.471791 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:53:14.471885 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.471811 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:53:14.472379 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.472358 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:14.477184 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.477159 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:53:14.484666 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.484648 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:14.485577 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485562 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:53:14.485647 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485603 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:53:14.485896 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485882 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:53:14.485963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485901 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:53:14.485963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485911 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:53:14.485963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485920 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:53:14.485963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485929 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:53:14.485963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.485938 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:53:14.486704 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486694 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:53:14.486755 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486708 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:53:14.486755 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486719 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:53:14.486755 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486730 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:53:14.486755 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486744 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:53:14.486902 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.486758 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:53:14.488366 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.488351 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:53:14.488414 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.488409 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:53:14.492571 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.492553 2577 server.go:1263] "Started kubelet" Oct 30 08:53:14.492691 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.492617 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:53:14.492747 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.492660 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:53:14.492808 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.492754 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:53:14.493382 ip-10-0-129-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:53:14.494605 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.494589 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:53:14.494699 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.494650 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:53:14.500039 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.500018 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 08:53:14.504321 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.504305 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:14.504908 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.504880 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:53:14.505275 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.505137 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:14.505742 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.505728 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:53:14.506002 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.505930 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:53:14.506002 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.505945 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:53:14.506130 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.506049 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:53:14.506130 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.506054 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:53:14.507306 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507285 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:53:14.507306 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507306 2577 factory.go:55] Registering systemd factory Oct 30 08:53:14.507450 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507316 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:53:14.507557 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507542 2577 factory.go:153] Registering CRI-O factory Oct 30 08:53:14.507557 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507559 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 08:53:14.507659 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507573 2577 factory.go:103] Registering Raw factory Oct 30 08:53:14.507659 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.507583 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 08:53:14.508527 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.508511 2577 manager.go:319] Starting recovery of all containers Oct 30 08:53:14.520175 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.520151 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:14.520645 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.520619 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 08:53:14.521883 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.520617 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-105.ec2.internal.18e342aea5889808 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-105.ec2.internal,},FirstTimestamp:2026-10-30 08:53:14.492524552 +0000 UTC m=+0.408838881,LastTimestamp:2026-10-30 08:53:14.492524552 +0000 UTC m=+0.408838881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-105.ec2.internal,}" Oct 30 08:53:14.523468 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.523452 2577 manager.go:324] Recovery completed Oct 30 08:53:14.527536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.527522 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.529388 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.529357 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.529462 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.529400 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.529462 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.529413 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.530059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530041 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:53:14.530059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530056 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:53:14.530156 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530076 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:53:14.530156 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530045 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.530942 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530927 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.531036 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530964 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.531036 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.530979 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.532257 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.532244 2577 policy_none.go:50] "Start" Oct 30 08:53:14.532299 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.532262 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:53:14.532299 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.532275 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:53:14.534241 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.534229 2577 policy_none.go:44] "Start" Oct 30 08:53:14.534862 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.534844 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 08:53:14.535424 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.535391 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 08:53:14.535481 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.535393 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 08:53:14.537611 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.537596 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knt9t" Oct 30 08:53:14.544810 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.544794 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knt9t" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.557632 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.557597 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-105.ec2.internal.18e342aea7bb08eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-105.ec2.internal,},FirstTimestamp:2026-10-30 08:53:14.529384683 +0000 UTC m=+0.445699011,LastTimestamp:2026-10-30 08:53:14.529384683 +0000 UTC m=+0.445699011,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-105.ec2.internal,}" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.572270 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.574217 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.574933 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.574967 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.574979 2577 server.go:87] "Starting device plugin registration server" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575248 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575258 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575444 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575518 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575526 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575615 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575642 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.575726 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.575818 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.576070 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:53:14.585623 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.576130 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:14.592178 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.592160 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:14.676275 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.676177 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.676912 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.676882 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal"] Oct 30 08:53:14.677035 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.676951 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.677209 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677179 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.677265 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677211 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.677265 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677223 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.677265 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677246 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.677686 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677663 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.677749 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677699 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.677749 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.677716 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.679915 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.679897 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.680054 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.680172 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680068 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.680550 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680532 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.680617 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680537 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.680617 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680585 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.680617 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680596 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.680617 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680562 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.680755 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.680631 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.682826 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.682809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.682911 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.682832 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:14.683899 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.683880 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:14.683979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.683911 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:14.683979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.683927 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:14.697315 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.697296 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.697393 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.697319 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-105.ec2.internal\": node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:14.707044 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.706885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.707044 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.706944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.710714 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.710694 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-105.ec2.internal\" not found" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.712977 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.712961 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-105.ec2.internal\" not found" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.734827 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.734798 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:14.807821 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.807793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.807979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.807826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.807979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.807843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.807979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.807882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.807979 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.807916 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.835691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.835648 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:14.908215 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.908183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.908373 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:14.908234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:14.936635 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:14.936577 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:15.016266 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:15.016236 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:15.018382 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:15.018367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:15.037639 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:15.037506 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:16.823634 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:16.823596 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:16.924135 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:16.924062 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:17.024636 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:17.024602 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:17.117117 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.117084 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 08:53:17.117592 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.117237 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:17.125076 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:17.125057 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:17.173343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.173314 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:17.190461 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.190403 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:17.191507 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.191488 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:17.216227 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.216201 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:17.218193 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.218177 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:17.236396 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.236339 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:16 +0000 UTC" deadline="2028-05-18 00:11:35.300431818 +0000 UTC" Oct 30 08:53:17.236396 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.236395 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13575h18m18.064040717s" Oct 30 08:53:17.238768 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.238753 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:17.240010 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.239986 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:17.240106 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.240059 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xrqqz" Oct 30 08:53:17.359470 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:17.359439 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba69e635f74a36f196d077f731ece3db.slice/crio-4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370 WatchSource:0}: Error finding container 4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370: Status 404 returned error can't find the container with id 4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370 Oct 30 08:53:17.359887 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:17.359870 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode421a574dfe363a6bf33baf4cfc6af78.slice/crio-b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800 WatchSource:0}: Error finding container b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800: Status 404 returned error can't find the container with id b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800 Oct 30 08:53:17.364581 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.364565 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:17.365038 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.365025 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:53:17.858038 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:17.858005 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:18.159244 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.158940 2577 apiserver.go:52] "Watching apiserver" Oct 30 08:53:18.201554 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.201518 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:53:18.202169 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.202145 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-xb9sq","openshift-multus/network-metrics-daemon-zqlwh","openshift-network-operator/iptables-alerter-pkr5v","openshift-cluster-node-tuning-operator/tuned-lzhf7","openshift-image-registry/node-ca-2zzzz","openshift-multus/multus-lvb22","kube-system/global-pull-secret-syncer-p9764","kube-system/konnectivity-agent-64jq8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg","openshift-ovn-kubernetes/ovnkube-node-86p9d","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal","openshift-dns/node-resolver-l6qzz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-5rc7p"] Oct 30 08:53:18.204963 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.204937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.205076 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.205028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:18.207107 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.207083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.209513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.209494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.211531 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.211511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.213613 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.213596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.216501 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.216388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.220624 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.219911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.222626 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.222603 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:53:18.222728 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.222705 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:53:18.222785 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.222738 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:53:18.222882 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.222869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:18.222961 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.222941 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:18.224110 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.223990 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.224596 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.224577 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.224677 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.224595 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:53:18.224677 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.224667 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:53:18.226042 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.226008 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.226301 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.226281 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:53:18.226406 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.226316 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.228677 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.228571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.228677 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.228652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:18.231339 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.230943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.231339 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.231282 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:53:18.231339 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.231334 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:53:18.231510 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.231367 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.231614 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.231591 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.233573 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.233431 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:53:18.233573 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.233490 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.233573 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.233506 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.233573 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.233546 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:53:18.233828 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.233694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.235623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.235608 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:53:18.235719 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.235699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.235785 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.235747 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:53:18.235842 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.235798 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.236822 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.236786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.239099 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.239080 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:53:18.243184 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.243157 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:17 +0000 UTC" deadline="2028-04-22 14:25:09.693137803 +0000 UTC" Oct 30 08:53:18.243184 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.243183 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12965h31m51.449957459s" Oct 30 08:53:18.268055 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.268002 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"ba69e635f74a36f196d077f731ece3db","Type":"ContainerStarted","Data":"4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370"} Oct 30 08:53:18.269250 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.269209 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerStarted","Data":"b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800"} Oct 30 08:53:18.293772 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.293750 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:53:18.303805 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.303931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.303931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.303931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.303931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.303931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304112 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304112 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.303970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwwt\" (UniqueName: \"kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304112 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.304112 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.304112 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6qg\" (UniqueName: \"kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9562\" (UniqueName: \"kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.304324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjdg\" (UniqueName: \"kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcc7\" (UniqueName: \"kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.304635 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmts\" (UniqueName: \"kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.305062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnnc\" (UniqueName: \"kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwx2\" (UniqueName: \"kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.304998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44spc\" (UniqueName: \"kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.305536 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.305980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.305466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.322700 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322631 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.322700 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322625 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:53:18.322700 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322670 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.322947 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322839 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:53:18.323050 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322950 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:53:18.323050 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322980 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:53:18.323050 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.322994 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:53:18.323050 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323036 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:53:18.323050 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323053 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:53:18.323337 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323108 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:53:18.323337 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323132 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:53:18.323337 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323156 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:53:18.323443 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.323345 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:53:18.406411 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406506 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.406585 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnnc\" (UniqueName: \"kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwx2\" (UniqueName: \"kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44spc\" (UniqueName: \"kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.406943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406944 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.406985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407154 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407252 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407540 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.407657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407669 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407701 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.407793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408147 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.408488 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.408574 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.408645 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:18.908622998 +0000 UTC m=+3.139396770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwwt\" (UniqueName: \"kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.409355 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6qg\" (UniqueName: \"kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9562\" (UniqueName: \"kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409451 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjdg\" (UniqueName: \"kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409644 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcc7\" (UniqueName: \"kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.410943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmts\" (UniqueName: \"kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409967 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.409184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410045 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410291 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.411715 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410833 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.410984 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411139 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411171 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411257 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411294 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.411457 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411479 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.411514 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:18.911489156 +0000 UTC m=+3.142262915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411627 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.411976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412054 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.412505 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.408596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412106 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412580 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.413312 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.412774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.414329 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.414308 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.416510 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.416491 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.416657 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.416638 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.428492 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.428416 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44spc\" (UniqueName: \"kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.428672 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.428651 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:18.428733 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.428683 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:18.428733 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.428697 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:18.428788 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.428753 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:18.928739126 +0000 UTC m=+3.159512914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:18.429788 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.429294 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnnc\" (UniqueName: \"kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.430652 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.430393 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwwt\" (UniqueName: \"kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.431230 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.431191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwx2\" (UniqueName: \"kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.433161 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.432572 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjdg\" (UniqueName: \"kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.433161 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.432582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6qg\" (UniqueName: \"kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.433161 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.432638 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcc7\" (UniqueName: \"kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.433161 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.432786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9562\" (UniqueName: \"kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.440210 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.440186 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmts\" (UniqueName: \"kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.523679 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.523640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:18.535044 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.535015 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a50c02c_9976_4034_8785_68c3e971060a.slice/crio-c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43 WatchSource:0}: Error finding container c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43: Status 404 returned error can't find the container with id c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43 Oct 30 08:53:18.536047 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.536025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:18.545849 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.545818 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65430e83_f633_41de_83c2_efc743b8f140.slice/crio-65ffe7af5b48fda9fdd21c4d752d03050d8c221cfeb285eb9e92d8dc009156f1 WatchSource:0}: Error finding container 65ffe7af5b48fda9fdd21c4d752d03050d8c221cfeb285eb9e92d8dc009156f1: Status 404 returned error can't find the container with id 65ffe7af5b48fda9fdd21c4d752d03050d8c221cfeb285eb9e92d8dc009156f1 Oct 30 08:53:18.546518 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.546496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:18.552899 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.552879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:18.560770 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.560749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvb22" Oct 30 08:53:18.573692 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.573653 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:18.581862 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.581835 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb312004_8950_4bea_ab8c_2c5279052edf.slice/crio-bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382 WatchSource:0}: Error finding container bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382: Status 404 returned error can't find the container with id bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382 Oct 30 08:53:18.592882 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.591589 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7411dc_7fed_43b4_be33_7eca7c3144d5.slice/crio-b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837 WatchSource:0}: Error finding container b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837: Status 404 returned error can't find the container with id b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837 Oct 30 08:53:18.593283 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.593240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:18.602300 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.602048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:18.604061 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.604037 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e96f32_b0b7_40b7_97e5_61ff980cc465.slice/crio-50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8 WatchSource:0}: Error finding container 50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8: Status 404 returned error can't find the container with id 50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8 Oct 30 08:53:18.607948 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.607925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:18.612181 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.612157 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d06a2c_a395_419e_b1d4_2285608d71da.slice/crio-a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911 WatchSource:0}: Error finding container a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911: Status 404 returned error can't find the container with id a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911 Oct 30 08:53:18.617545 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:18.617523 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a1416c_5f6d_4ac7_9f38_2760a05daa83.slice/crio-8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1 WatchSource:0}: Error finding container 8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1: Status 404 returned error can't find the container with id 8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1 Oct 30 08:53:18.720193 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.720103 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:18.913430 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.913394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:18.913586 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:18.913452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:18.913586 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.913564 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:18.913684 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.913636 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:19.91361747 +0000 UTC m=+4.144391238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:18.913684 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.913564 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:18.913769 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:18.913719 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:19.913706567 +0000 UTC m=+4.144480344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:19.014304 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.014211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:19.014437 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.014365 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:19.014437 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.014381 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:19.014437 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.014392 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:19.014564 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.014446 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:20.014429271 +0000 UTC m=+4.245203042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:19.245239 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.243553 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:17 +0000 UTC" deadline="2028-05-08 14:31:29.100159748 +0000 UTC" Oct 30 08:53:19.245239 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.243609 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13349h38m9.856554667s" Oct 30 08:53:19.281826 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.281005 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"ba69e635f74a36f196d077f731ece3db","Type":"ContainerStarted","Data":"341fd64337799d618e7d771e4cd20eedd72be6772d7fd50b38d9086f5a6224a6"} Oct 30 08:53:19.286059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.285977 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerStarted","Data":"8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1"} Oct 30 08:53:19.287640 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.287544 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911"} Oct 30 08:53:19.291335 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.291226 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" event={"ID":"8f7411dc-7fed-43b4-be33-7eca7c3144d5","Type":"ContainerStarted","Data":"b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837"} Oct 30 08:53:19.292769 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.292742 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvb22" event={"ID":"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49","Type":"ContainerStarted","Data":"8130978ab436abea23c76b7105e5242296011d95be0797c43e231396d1909880"} Oct 30 08:53:19.294451 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.294423 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zzzz" event={"ID":"8d2f4c5d-a352-4c80-8794-edf020ddbafa","Type":"ContainerStarted","Data":"81aa7b94185e8fee5ca23b9951b10815e719e40dbb678f9864b4a7b7c1a3fb75"} Oct 30 08:53:19.295793 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.295770 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr5v" event={"ID":"fb312004-8950-4bea-ab8c-2c5279052edf","Type":"ContainerStarted","Data":"bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382"} Oct 30 08:53:19.299890 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.299864 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" event={"ID":"65430e83-f633-41de-83c2-efc743b8f140","Type":"ContainerStarted","Data":"65ffe7af5b48fda9fdd21c4d752d03050d8c221cfeb285eb9e92d8dc009156f1"} Oct 30 08:53:19.307690 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.307283 2577 generic.go:356] "Generic (PLEG): container finished" podID="e421a574dfe363a6bf33baf4cfc6af78" containerID="05796485adf348412b79a9cc895a2c916d40f0171af94c1ce26690764f60ed3a" exitCode=0 Oct 30 08:53:19.307690 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.307360 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerDied","Data":"05796485adf348412b79a9cc895a2c916d40f0171af94c1ce26690764f60ed3a"} Oct 30 08:53:19.321801 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.312975 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6qzz" event={"ID":"61e96f32-b0b7-40b7-97e5-61ff980cc465","Type":"ContainerStarted","Data":"50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8"} Oct 30 08:53:19.325571 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.322303 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64jq8" event={"ID":"0a50c02c-9976-4034-8785-68c3e971060a","Type":"ContainerStarted","Data":"c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43"} Oct 30 08:53:19.363792 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.363527 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" podStartSLOduration=2.363503567 podStartE2EDuration="2.363503567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:19.321193132 +0000 UTC m=+3.551966907" watchObservedRunningTime="2026-10-30 08:53:19.363503567 +0000 UTC m=+3.594277342" Oct 30 08:53:19.922756 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.922721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:19.922943 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:19.922814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:19.923002 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.922946 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:19.923048 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.923005 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:21.922986009 +0000 UTC m=+6.153759784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:19.923495 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.923476 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:19.923570 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:19.923535 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:21.92352028 +0000 UTC m=+6.154294035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:20.024805 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:20.024761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:20.024978 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.024959 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:20.025038 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.024984 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:20.025038 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.024998 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:20.025139 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.025066 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:22.025046269 +0000 UTC m=+6.255820105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:20.263413 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:20.263337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:20.263855 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.263471 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:20.263921 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:20.263875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:20.264038 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.263967 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:20.264457 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:20.264285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:20.264457 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:20.264405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:20.331528 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:20.331488 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerStarted","Data":"f8fa46300fa33afd16a35915fbeb2e7aefaca90ee8851c8485f755309b74428b"} Oct 30 08:53:21.346338 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:21.344122 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerStarted","Data":"a885fc789c894502d1c51b207982f4a82bc605d68d7b8a11aa715abc07803765"} Oct 30 08:53:21.416531 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:21.416467 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" podStartSLOduration=4.416447879 podStartE2EDuration="4.416447879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:20.371309172 +0000 UTC m=+4.602082965" watchObservedRunningTime="2026-10-30 08:53:21.416447879 +0000 UTC m=+5.647221655" Oct 30 08:53:21.941114 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:21.941074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:21.941277 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:21.941180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:21.941352 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:21.941319 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:21.941399 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:21.941388 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:25.941370209 +0000 UTC m=+10.172143981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:21.941827 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:21.941807 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:21.941893 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:21.941866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:25.941850873 +0000 UTC m=+10.172624647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:22.041491 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.041449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:22.041666 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.041645 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:22.041726 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.041670 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:22.041726 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.041682 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:22.041826 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.041739 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:26.041721498 +0000 UTC m=+10.272495266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:22.273062 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.273031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:22.273236 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.273158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:22.273596 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.273544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:22.273693 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.273633 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:22.273693 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.273683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:22.273805 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:22.273758 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:22.349851 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.349815 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64jq8" event={"ID":"0a50c02c-9976-4034-8785-68c3e971060a","Type":"ContainerStarted","Data":"a4d3fdd743aba8836b4d86578e657c1886c8c56a8e7e43688f2273b086bbd062"} Oct 30 08:53:22.353436 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.352827 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="a885fc789c894502d1c51b207982f4a82bc605d68d7b8a11aa715abc07803765" exitCode=0 Oct 30 08:53:22.353436 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.352868 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"a885fc789c894502d1c51b207982f4a82bc605d68d7b8a11aa715abc07803765"} Oct 30 08:53:22.400493 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:22.399542 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-64jq8" podStartSLOduration=2.882911885 podStartE2EDuration="6.399524584s" totalImagesPullingTime="3.516612699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:22.399466839 +0000 UTC m=+6.630240629" watchObservedRunningTime="2026-10-30 08:53:22.399524584 +0000 UTC m=+6.630298362" Oct 30 08:53:24.264208 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:24.264157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:24.264638 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:24.264295 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:24.264638 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:24.264364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:24.264638 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:24.264426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:24.264638 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:24.264482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:24.264638 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:24.264550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:24.359343 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:24.359190 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" event={"ID":"8f7411dc-7fed-43b4-be33-7eca7c3144d5","Type":"ContainerStarted","Data":"dde4b20945aff1a3d28374c8be9f8288c1274e14723379f4c5a90fd7833fa596"} Oct 30 08:53:24.363442 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:24.363389 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zzzz" event={"ID":"8d2f4c5d-a352-4c80-8794-edf020ddbafa","Type":"ContainerStarted","Data":"c754f891a5dd01857c41ca551702b0f00e070d72f3cae559c81dc4354ee1506f"} Oct 30 08:53:25.973645 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:25.973619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:25.974132 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:25.973670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:25.974206 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:25.973842 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:25.974336 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:25.973878 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:25.974336 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:25.974238 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:33.974204634 +0000 UTC m=+18.204978389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:25.974336 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:25.974312 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:33.974281789 +0000 UTC m=+18.205055549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:25.991484 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:25.991456 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:53:26.074948 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.074919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:26.075097 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.075080 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:26.075160 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.075107 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:26.075160 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.075120 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:26.075282 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.075178 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:34.075161078 +0000 UTC m=+18.305934830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:26.264004 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.263969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:26.264177 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.264099 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:26.264177 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.264168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:26.264322 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.264304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:26.264410 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.264387 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:26.264481 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:26.264429 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:26.266453 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.266357 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:53:25.991477888Z","UUID":"7afa0028-6b7d-4aea-96f7-62277086107e","Handler":null,"Name":"","Endpoint":""} Oct 30 08:53:26.272611 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.272586 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:53:26.272722 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.272634 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:53:26.373325 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.373286 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" event={"ID":"8f7411dc-7fed-43b4-be33-7eca7c3144d5","Type":"ContainerStarted","Data":"ecb7860585003e4f4dd2695d68910681a38977437f1066ff7f2479f2e6d08415"} Oct 30 08:53:26.377803 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.377773 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvb22" event={"ID":"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49","Type":"ContainerStarted","Data":"5b07d1f7916bd15d65d22f2166a144a82e810f48eddf20919bbeb702dca7a015"} Oct 30 08:53:26.380674 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.380645 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6qzz" event={"ID":"61e96f32-b0b7-40b7-97e5-61ff980cc465","Type":"ContainerStarted","Data":"4b30d4bd5e70f086c6f7e5f615a701ef34b77fad462431459093390960d61e76"} Oct 30 08:53:26.417655 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.417584 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2zzzz" podStartSLOduration=5.676657848 podStartE2EDuration="10.417543478s" totalImagesPullingTime="4.74088563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:24.40315807 +0000 UTC m=+8.633931844" watchObservedRunningTime="2026-10-30 08:53:26.417543478 +0000 UTC m=+10.648317249" Oct 30 08:53:26.457468 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:26.457200 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lvb22" podStartSLOduration=2.987204373 podStartE2EDuration="10.457180659s" totalImagesPullingTime="7.469976286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:26.418997857 +0000 UTC m=+10.649771635" watchObservedRunningTime="2026-10-30 08:53:26.457180659 +0000 UTC m=+10.687954435" Oct 30 08:53:27.264967 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.264700 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:27.265407 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.265295 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:27.303324 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.303263 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l6qzz" podStartSLOduration=3.731510498 podStartE2EDuration="11.303209987s" totalImagesPullingTime="7.571699489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:26.457455034 +0000 UTC m=+10.688228810" watchObservedRunningTime="2026-10-30 08:53:27.303209987 +0000 UTC m=+11.533983760" Oct 30 08:53:27.384911 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.384874 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="1d72d039d7bfdaa56c4c4af73fa0674e9d7c27265fa991758c90253471da9ac1" exitCode=0 Oct 30 08:53:27.385083 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.384953 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"1d72d039d7bfdaa56c4c4af73fa0674e9d7c27265fa991758c90253471da9ac1"} Oct 30 08:53:27.389209 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.389187 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" event={"ID":"8f7411dc-7fed-43b4-be33-7eca7c3144d5","Type":"ContainerStarted","Data":"9d8a9f060af4d9b70a58b324d6cd65dc38b8f6d1f66d8b26433738c1ff4c3304"} Oct 30 08:53:27.472089 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:27.472033 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" podStartSLOduration=3.571768404 podStartE2EDuration="11.472013242s" totalImagesPullingTime="7.900244838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:27.471408164 +0000 UTC m=+11.702181939" watchObservedRunningTime="2026-10-30 08:53:27.472013242 +0000 UTC m=+11.702787018" Oct 30 08:53:28.261615 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.261589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:28.261738 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.261645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:28.261738 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.261660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:28.261848 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:28.261738 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:28.261980 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:28.261838 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:28.261980 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:28.261943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:28.391661 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.391617 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr5v" event={"ID":"fb312004-8950-4bea-ab8c-2c5279052edf","Type":"ContainerStarted","Data":"bfaa0f2f7c398ebcddae239f945b2c8c7283a8b1a774f884e6ed8da2471c2ca1"} Oct 30 08:53:28.393284 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.393256 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" event={"ID":"65430e83-f633-41de-83c2-efc743b8f140","Type":"ContainerStarted","Data":"bb719a30578b9a0740b7f85a2d1c8d8eca7e03f6a0b8b332e9daddaf5a2b1bbe"} Oct 30 08:53:28.395514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.395490 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="4622c349f627806aa4de03021c0b10fba80eb81ee04b67a9a7994371821bafa8" exitCode=0 Oct 30 08:53:28.395605 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.395576 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"4622c349f627806aa4de03021c0b10fba80eb81ee04b67a9a7994371821bafa8"} Oct 30 08:53:28.430707 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.430647 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkr5v" podStartSLOduration=4.960198591 podStartE2EDuration="12.43063159s" totalImagesPullingTime="7.470432999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:28.430239865 +0000 UTC m=+12.661013643" watchObservedRunningTime="2026-10-30 08:53:28.43063159 +0000 UTC m=+12.661405365" Oct 30 08:53:28.513341 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:28.513241 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" podStartSLOduration=2.873086727 podStartE2EDuration="12.51320271s" totalImagesPullingTime="9.640115983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:28.512541411 +0000 UTC m=+12.743315187" watchObservedRunningTime="2026-10-30 08:53:28.51320271 +0000 UTC m=+12.743976487" Oct 30 08:53:29.400647 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:29.400607 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="64dcd806ed1f22b60ffeff15ac308b78c6df8e86bd5bcaa440c87e5154cbfb61" exitCode=0 Oct 30 08:53:29.401114 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:29.400675 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"64dcd806ed1f22b60ffeff15ac308b78c6df8e86bd5bcaa440c87e5154cbfb61"} Oct 30 08:53:30.261987 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:30.261957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:30.261987 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:30.261988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:30.262182 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:30.262060 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:30.262182 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:30.261957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:30.262182 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:30.262171 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:30.262314 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:30.262242 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:31.409130 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:31.408890 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="080f002500c1ee10c9d30aa39897f1c61a8fc085db09732c2ea2f6c3ef44e620" exitCode=0 Oct 30 08:53:31.409627 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:31.408940 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"080f002500c1ee10c9d30aa39897f1c61a8fc085db09732c2ea2f6c3ef44e620"} Oct 30 08:53:32.261699 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:32.261668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:32.261699 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:32.261689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:32.261917 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:32.261837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:32.261917 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:32.261877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:32.261982 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:32.261955 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:32.262040 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:32.262025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:32.414308 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:32.414273 2577 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="2d65f32d730c349e3f3413de77efb876009236b270b2303c0c5f5a5cdea12ff3" exitCode=0 Oct 30 08:53:32.414730 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:32.414328 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerDied","Data":"2d65f32d730c349e3f3413de77efb876009236b270b2303c0c5f5a5cdea12ff3"} Oct 30 08:53:33.420091 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.420053 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" event={"ID":"d0a1416c-5f6d-4ac7-9f38-2760a05daa83","Type":"ContainerStarted","Data":"e1f3be919b26131422ed5eb9d18f99db51e821510246ae65a98e0cf8cc3d9c2c"} Oct 30 08:53:33.423010 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.422987 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"b54fb9b3796ba4d992a86475aac2acafb817551a363b83a24ecc93d9bb89b1bf"} Oct 30 08:53:33.423126 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.423051 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"fb5c5a3a91cee5bec03d786436c89795aa30fcda831bdd617437d7aea7a627b2"} Oct 30 08:53:33.423126 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.423083 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"092e8de7da72e2481306fbad20f7b1c3bb3e01f73772c0397f1652bac6d0b740"} Oct 30 08:53:33.423126 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.423117 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"8660b06894aaecc8e5542afde9e2370694c51709d6ff48b39355cb9a140eaae1"} Oct 30 08:53:33.423236 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.423145 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"3c05a06d9007d48878cca9fd09202b76ca9148cdd017cec5c5af290beda410f0"} Oct 30 08:53:33.423236 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.423172 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"b9ee4e8dde580d2bb888f47fe28baa4df338864aa2dbc9eba143cb1a36e2c448"} Oct 30 08:53:33.462811 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:33.462774 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5rc7p" podStartSLOduration=8.213965823 podStartE2EDuration="17.462762659s" totalImagesPullingTime="9.248796836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:33.462068924 +0000 UTC m=+17.692842703" watchObservedRunningTime="2026-10-30 08:53:33.462762659 +0000 UTC m=+17.693536433" Oct 30 08:53:34.035636 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.035597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:34.035831 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.035655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:34.035831 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.035747 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:34.035831 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.035793 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:50.035781205 +0000 UTC m=+34.266554970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:34.035831 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.035747 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:34.035962 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.035869 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret podName:7d7f6695-43a2-4f59-83c5-ee24748be7f5 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:50.03585604 +0000 UTC m=+34.266629792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret") pod "global-pull-secret-syncer-p9764" (UID: "7d7f6695-43a2-4f59-83c5-ee24748be7f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:34.136603 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.136562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:34.136765 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.136690 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:34.136765 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.136706 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:34.136765 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.136714 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9vf5k for pod openshift-network-diagnostics/network-check-target-xb9sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:34.136765 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.136761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k podName:b00097a4-0c42-452a-af4f-041cf40aecab nodeName:}" failed. No retries permitted until 2026-10-30 08:53:50.136748209 +0000 UTC m=+34.367521974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9vf5k" (UniqueName: "kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k") pod "network-check-target-xb9sq" (UID: "b00097a4-0c42-452a-af4f-041cf40aecab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:34.261859 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.261821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:34.262010 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.261830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:34.262010 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:34.261963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:34.262010 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.261958 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:34.262122 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.262024 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:34.262122 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:34.262078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:35.430147 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:35.430052 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"c1d6fba079a9565f2362328b9d0ae7afb9f83a2b97433a33855fd8075cc84cfd"} Oct 30 08:53:36.262565 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:36.262532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:36.262775 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:36.262682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:36.262775 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:36.262696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:36.262896 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:36.262786 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:36.262896 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:36.262843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:36.262997 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:36.262923 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:37.437629 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:37.437547 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" event={"ID":"95d06a2c-a395-419e-b1d4-2285608d71da","Type":"ContainerStarted","Data":"eaade5310dd2f8837867a424f409528112c0bfa4f756f91d32dbfde2dea66798"} Oct 30 08:53:37.438489 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:37.437756 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:37.438489 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:37.437819 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:37.452843 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:37.452818 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:37.522726 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:37.522676 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" podStartSLOduration=7.421389755 podStartE2EDuration="21.522655714s" totalImagesPullingTime="14.101265959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:37.522268524 +0000 UTC m=+21.753042297" watchObservedRunningTime="2026-10-30 08:53:37.522655714 +0000 UTC m=+21.753429487" Oct 30 08:53:38.262340 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:38.262307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:38.262340 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:38.262320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:38.262579 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:38.262327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:38.262579 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:38.262426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:38.262579 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:38.262479 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:38.262579 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:38.262565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:38.440499 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:38.440467 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:38.460029 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:38.460005 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:39.266013 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.265842 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xb9sq"] Oct 30 08:53:39.266181 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.266091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:39.266266 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:39.266190 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:39.269208 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.269187 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqlwh"] Oct 30 08:53:39.269338 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.269300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:39.269429 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:39.269412 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:39.269683 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.269664 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p9764"] Oct 30 08:53:39.269752 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:39.269741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:39.269821 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:39.269806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:40.905012 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:40.904979 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:40.905565 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:40.905117 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:40.905652 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:40.905636 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:41.262401 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:41.262370 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:41.262549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:41.262381 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:41.262549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:41.262384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:41.262660 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:41.262536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:41.262660 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:41.262622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:41.262765 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:41.262704 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:43.261601 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:43.261573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:43.261601 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:43.261591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:43.262036 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:43.261603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:43.262036 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:43.261733 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zqlwh" podUID="0ff1f68b-203c-43a3-b5a5-671eda399b32" Oct 30 08:53:43.262036 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:43.261834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9764" podUID="7d7f6695-43a2-4f59-83c5-ee24748be7f5" Oct 30 08:53:43.262036 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:43.261917 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb9sq" podUID="b00097a4-0c42-452a-af4f-041cf40aecab" Oct 30 08:53:44.622180 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.622157 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeReady" Oct 30 08:53:44.622627 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.622281 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 08:53:44.662788 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.662756 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccr66"] Oct 30 08:53:44.665855 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.665829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:44.666623 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.666601 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8lfmm"] Oct 30 08:53:44.669811 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.669792 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hhlxw"] Oct 30 08:53:44.669907 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.669892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:44.672837 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.672815 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4p7wn"] Oct 30 08:53:44.672936 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.672919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:44.675831 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.675808 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:53:44.675946 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.675928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.682240 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.682153 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq"] Oct 30 08:53:44.682558 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.682536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.685293 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.685235 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2"] Oct 30 08:53:44.685691 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.685659 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.688419 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.688396 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.688513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.688451 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.689586 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.689567 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mxzf"] Oct 30 08:53:44.690148 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.690133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.694379 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.694356 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr"] Oct 30 08:53:44.694518 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.694478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.697123 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.697106 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 08:53:44.697335 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.697321 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rm2lv\"" Oct 30 08:53:44.697469 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.697456 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tn25f"] Oct 30 08:53:44.697568 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.697555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.702010 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.701988 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:53:44.702107 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.702093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.706297 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.706279 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z"] Oct 30 08:53:44.706766 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.706746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.710621 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.709795 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7888d9c8bc-nzrhq"] Oct 30 08:53:44.712382 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.712040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.714499 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.714479 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4"] Oct 30 08:53:44.715344 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.715321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.717262 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.717243 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-477hl"] Oct 30 08:53:44.717408 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.717394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:44.722052 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.720998 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mj9w5"] Oct 30 08:53:44.722052 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.721660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:44.724189 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.724166 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vbc5k"] Oct 30 08:53:44.725093 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.725069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728837 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccr66"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728884 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8lfmm"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728906 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hhlxw"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728933 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4p7wn"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728959 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.728985 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729007 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mxzf"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729033 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729062 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tn25f"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729089 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-477hl"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729112 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729145 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.729763 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.729466 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4"] Oct 30 08:53:44.734476 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.731848 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbc5k"] Oct 30 08:53:44.735234 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.735197 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mj9w5"] Oct 30 08:53:44.740076 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.740055 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7888d9c8bc-nzrhq"] Oct 30 08:53:44.742996 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.742977 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:53:44.789552 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789531 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.789761 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789733 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.789761 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789748 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789770 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789788 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789799 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789753 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789806 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.789895 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789883 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789917 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789981 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.789987 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790033 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790080 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 08:53:44.790173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790165 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790182 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790202 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790232 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790255 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790284 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790306 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790324 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790329 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790335 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.790513 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790374 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 08:53:44.790988 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790546 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:44.790988 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.790630 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 08:53:44.791079 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.791051 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 08:53:44.816475 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.816629 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.816629 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.816629 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.816789 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8xk\" (UniqueName: \"kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.816789 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.816789 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw65l\" (UniqueName: \"kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.816789 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.816789 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816890 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816892 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816913 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816939 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816964 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816978 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 08:53:44.817006 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.816972 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817088 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817118 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817170 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt7w\" (UniqueName: \"kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w\") pod \"network-check-source-6f78d59f99-477hl\" (UID: \"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817336 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817343 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44jz\" (UniqueName: \"kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsh6f\" (UniqueName: \"kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f\") pod \"volume-data-source-validator-58c9f5f46d-8jhj4\" (UID: \"412a8656-9cb2-4e67-9752-f179cd02219b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817402 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.817534 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxwg\" (UniqueName: \"kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg\") pod \"kas-connection-checker-685895c7c9-ccr66\" (UID: \"0d9c9c95-575e-4d45-9416-2c8bd643a5e3\") " pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7r7\" (UniqueName: \"kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.817974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.818059 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6q6\" (UniqueName: \"kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhcb\" (UniqueName: \"kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898q4\" (UniqueName: \"kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwk7n\" (UniqueName: \"kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n\") pod \"kas-connection-checker-685895c7c9-hhlxw\" (UID: \"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169\") " pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.818514 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h26\" (UniqueName: \"kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26\") pod \"kas-connection-checker-685895c7c9-8lfmm\" (UID: \"627ed97e-e994-4439-815e-d7fe229402e5\") " pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:44.818993 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.818993 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.818993 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.818993 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.818993 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.818653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.827928 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.827905 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:53:44.827928 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.827916 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:53:44.828881 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.828862 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m8kth\"" Oct 30 08:53:44.828940 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.828918 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wkc6q\"" Oct 30 08:53:44.828940 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.828932 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c7tn4\"" Oct 30 08:53:44.829026 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.828964 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4wbtr\"" Oct 30 08:53:44.829321 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.829308 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dfndm\"" Oct 30 08:53:44.829674 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.829657 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w9gxh\"" Oct 30 08:53:44.830106 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.830090 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 08:53:44.830434 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.830419 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 08:53:44.830975 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.830956 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 08:53:44.831520 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.831507 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 08:53:44.841785 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.841764 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 08:53:44.919931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.919896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.919931 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.919933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.920363 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.919957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8xk\" (UniqueName: \"kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.920363 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.919980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.920363 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw65l\" (UniqueName: \"kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.920363 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.920589 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt7w\" (UniqueName: \"kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w\") pod \"network-check-source-6f78d59f99-477hl\" (UID: \"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920679 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44jz\" (UniqueName: \"kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsh6f\" (UniqueName: \"kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f\") pod \"volume-data-source-validator-58c9f5f46d-8jhj4\" (UID: \"412a8656-9cb2-4e67-9752-f179cd02219b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.920779 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxwg\" (UniqueName: \"kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg\") pod \"kas-connection-checker-685895c7c9-ccr66\" (UID: \"0d9c9c95-575e-4d45-9416-2c8bd643a5e3\") " pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.920835 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.42081728 +0000 UTC m=+29.651591036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:44.920903 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.920905 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921281 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921298 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65ffb76754-ndfbg: secret "image-registry-tls" not found Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921351 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls podName:c86dd71b-65b4-48bc-9b09-772f6dd7053a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.421334182 +0000 UTC m=+29.652107958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls") pod "image-registry-65ffb76754-ndfbg" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a") : secret "image-registry-tls" not found Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921409 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921419 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fd88cff75-jmrdx: secret "image-registry-tls" not found Oct 30 08:53:44.921466 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921452 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls podName:25c4713f-3616-4d01-b7c7-310f70a14127 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.421441532 +0000 UTC m=+29.652215305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls") pod "image-registry-fd88cff75-jmrdx" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127") : secret "image-registry-tls" not found Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7r7\" (UniqueName: \"kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921669 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921717 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 08:53:44.921905 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.921903 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.421756738 +0000 UTC m=+29.652530503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : secret "router-metrics-certs-default" not found Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6q6\" (UniqueName: \"kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhcb\" (UniqueName: \"kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.922412 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.922849 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922490 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.922849 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.922785 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.923440 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.923440 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923326 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.923440 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.923633 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.923682 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.923682 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.923682 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.921987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.923880 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.924134 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.924458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.924893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.924928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.924954 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.924937 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.925350 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.925350 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.925144 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:44.925350 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898q4\" (UniqueName: \"kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.925350 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.925310 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls podName:c9e051aa-16cd-43b2-aec2-4f19f35ab485 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.425294684 +0000 UTC m=+29.656068440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls") pod "dns-default-mj9w5" (UID: "c9e051aa-16cd-43b2-aec2-4f19f35ab485") : secret "dns-default-metrics-tls" not found Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk7n\" (UniqueName: \"kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n\") pod \"kas-connection-checker-685895c7c9-hhlxw\" (UID: \"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169\") " pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h26\" (UniqueName: \"kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26\") pod \"kas-connection-checker-685895c7c9-8lfmm\" (UID: \"627ed97e-e994-4439-815e-d7fe229402e5\") " pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.925976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926086 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.926397 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.926244 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.926290 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.426276857 +0000 UTC m=+29.657050623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.926328 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.926381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.426363606 +0000 UTC m=+29.657137375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926515 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:44.926685 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:45.426670492 +0000 UTC m=+29.657444264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926792 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:44.927057 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.926890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.927305 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.927205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.928151 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.928122 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.928272 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.928253 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.928619 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.928605 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.929259 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.929237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.942917 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.942840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8xk\" (UniqueName: \"kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:44.943491 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.943466 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.943841 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.943816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw65l\" (UniqueName: \"kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:44.943988 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.943914 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44jz\" (UniqueName: \"kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:44.945018 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.944989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.946204 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.945606 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxwg\" (UniqueName: \"kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg\") pod \"kas-connection-checker-685895c7c9-ccr66\" (UID: \"0d9c9c95-575e-4d45-9416-2c8bd643a5e3\") " pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:44.946204 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.945668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt7w\" (UniqueName: \"kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w\") pod \"network-check-source-6f78d59f99-477hl\" (UID: \"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:44.946204 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.945804 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsh6f\" (UniqueName: \"kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f\") pod \"volume-data-source-validator-58c9f5f46d-8jhj4\" (UID: \"412a8656-9cb2-4e67-9752-f179cd02219b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:44.946204 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.946141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7r7\" (UniqueName: \"kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:44.946204 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.946165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:44.946528 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.946481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6q6\" (UniqueName: \"kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:44.947748 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.947726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:44.947938 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.947908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-898q4\" (UniqueName: \"kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:44.948029 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.947925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h26\" (UniqueName: \"kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26\") pod \"kas-connection-checker-685895c7c9-8lfmm\" (UID: \"627ed97e-e994-4439-815e-d7fe229402e5\") " pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:44.948090 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.948074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:44.948299 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.948279 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhcb\" (UniqueName: \"kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:44.948673 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.948658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk7n\" (UniqueName: \"kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n\") pod \"kas-connection-checker-685895c7c9-hhlxw\" (UID: \"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169\") " pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:44.948765 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.948749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:44.978584 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.978559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:44.990608 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.990476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:44.997047 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:44.997007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:45.061142 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.057550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:45.067416 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.064821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:45.074600 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.069040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:45.080723 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.080530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:45.086651 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.086618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:45.093496 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.093397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:45.103612 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.103565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:45.262286 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.262260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:45.262435 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.262295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:45.262435 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.262341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:45.277444 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.277424 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:53:45.277559 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.277517 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:53:45.277693 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.277680 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:53:45.277693 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.277683 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:53:45.277791 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.277735 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:53:45.289774 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.289575 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccr66"] Oct 30 08:53:45.291728 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.291584 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9c9c95_575e_4d45_9416_2c8bd643a5e3.slice/crio-bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083 WatchSource:0}: Error finding container bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083: Status 404 returned error can't find the container with id bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083 Oct 30 08:53:45.292622 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.292590 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8lfmm"] Oct 30 08:53:45.295281 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.294936 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627ed97e_e994_4439_815e_d7fe229402e5.slice/crio-8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861 WatchSource:0}: Error finding container 8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861: Status 404 returned error can't find the container with id 8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861 Oct 30 08:53:45.295731 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.295666 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hhlxw"] Oct 30 08:53:45.296911 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.296886 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9b72a7_a9b7_4fbb_b3ec_248a6f95c169.slice/crio-969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e WatchSource:0}: Error finding container 969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e: Status 404 returned error can't find the container with id 969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e Oct 30 08:53:45.399209 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.399155 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbbc457_38a1_45fd_ab3d_738f7dd07913.slice/crio-b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb WatchSource:0}: Error finding container b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb: Status 404 returned error can't find the container with id b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb Oct 30 08:53:45.411871 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.411850 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mxzf"] Oct 30 08:53:45.422697 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.422669 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z"] Oct 30 08:53:45.425357 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.425280 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tn25f"] Oct 30 08:53:45.430777 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:45.430777 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:45.430777 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:45.430934 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:45.430934 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:45.430934 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.430869 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.430848681 +0000 UTC m=+30.661622439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:45.430934 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.430928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:45.431140 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431061 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:45.431140 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.431071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:45.431140 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431065 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:45.431305 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431122 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:45.431305 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431157 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fd88cff75-jmrdx: secret "image-registry-tls" not found Oct 30 08:53:45.431305 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431168 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 08:53:45.431477 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431316 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:45.431477 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431326 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65ffb76754-ndfbg: secret "image-registry-tls" not found Oct 30 08:53:45.431477 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431393 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431379414 +0000 UTC m=+30.662153165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:45.431477 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431407 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431400919 +0000 UTC m=+30.662174670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:45.431477 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431417 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls podName:25c4713f-3616-4d01-b7c7-310f70a14127 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431411848 +0000 UTC m=+30.662185600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls") pod "image-registry-fd88cff75-jmrdx" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127") : secret "image-registry-tls" not found Oct 30 08:53:45.431736 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431482 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls podName:c86dd71b-65b4-48bc-9b09-772f6dd7053a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431475947 +0000 UTC m=+30.662249699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls") pod "image-registry-65ffb76754-ndfbg" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a") : secret "image-registry-tls" not found Oct 30 08:53:45.431736 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.431554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:45.431736 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431589 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431580358 +0000 UTC m=+30.662354110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : secret "router-metrics-certs-default" not found Oct 30 08:53:45.431736 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431700 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:45.431889 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.431775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls podName:c9e051aa-16cd-43b2-aec2-4f19f35ab485 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.431762371 +0000 UTC m=+30.662536123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls") pod "dns-default-mj9w5" (UID: "c9e051aa-16cd-43b2-aec2-4f19f35ab485") : secret "dns-default-metrics-tls" not found Oct 30 08:53:45.433250 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.432244 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:45.433250 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:45.432319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:46.432303437 +0000 UTC m=+30.663077191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:45.440286 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.440263 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4"] Oct 30 08:53:45.442312 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.442283 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412a8656_9cb2_4e67_9752_f179cd02219b.slice/crio-5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4 WatchSource:0}: Error finding container 5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4: Status 404 returned error can't find the container with id 5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4 Oct 30 08:53:45.443717 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.443697 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbc5k"] Oct 30 08:53:45.445723 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.445702 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e00d62_c126_48fb_aab2_d1eac679b710.slice/crio-46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97 WatchSource:0}: Error finding container 46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97: Status 404 returned error can't find the container with id 46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97 Oct 30 08:53:45.455844 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.455071 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr"] Oct 30 08:53:45.459036 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:45.459015 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae3e225_68cf_4814_8d33_e184ad008b3c.slice/crio-7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14 WatchSource:0}: Error finding container 7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14: Status 404 returned error can't find the container with id 7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14 Oct 30 08:53:45.460707 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.460684 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-477hl"] Oct 30 08:53:45.460858 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.460837 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerStarted","Data":"b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb"} Oct 30 08:53:45.465687 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.465659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" event={"ID":"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169","Type":"ContainerStarted","Data":"969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e"} Oct 30 08:53:45.470002 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.469972 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" event={"ID":"47f381ca-b571-4170-a89c-ecdef16c72f6","Type":"ContainerStarted","Data":"3c5915954fd7b6e75d6d90931cc3576246b848260928f36183513c19ff19dceb"} Oct 30 08:53:45.471289 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.471273 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ccr66" event={"ID":"0d9c9c95-575e-4d45-9416-2c8bd643a5e3","Type":"ContainerStarted","Data":"bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083"} Oct 30 08:53:45.474014 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.473969 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tn25f" event={"ID":"8f93bd2e-a21e-4bc1-993d-117adc818868","Type":"ContainerStarted","Data":"e2dba5842d55c7692b989399bb119b6f5eaec878271dcbdf3db36f7b3b34b203"} Oct 30 08:53:45.475178 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.475153 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" event={"ID":"627ed97e-e994-4439-815e-d7fe229402e5","Type":"ContainerStarted","Data":"8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861"} Oct 30 08:53:45.476886 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.476512 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" event={"ID":"412a8656-9cb2-4e67-9752-f179cd02219b","Type":"ContainerStarted","Data":"5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4"} Oct 30 08:53:45.478036 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:45.477998 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbc5k" event={"ID":"11e00d62-c126-48fb-aab2-d1eac679b710","Type":"ContainerStarted","Data":"46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97"} Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:46.456022 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.455742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:46.457226 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457189 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457206 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457267 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457259427 +0000 UTC m=+32.688033194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457284 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65ffb76754-ndfbg: secret "image-registry-tls" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457299 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls podName:c9e051aa-16cd-43b2-aec2-4f19f35ab485 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457290356 +0000 UTC m=+32.688064110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls") pod "dns-default-mj9w5" (UID: "c9e051aa-16cd-43b2-aec2-4f19f35ab485") : secret "dns-default-metrics-tls" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls podName:c86dd71b-65b4-48bc-9b09-772f6dd7053a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457311018 +0000 UTC m=+32.688084783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls") pod "image-registry-65ffb76754-ndfbg" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a") : secret "image-registry-tls" not found Oct 30 08:53:46.457344 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457344 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457326682 +0000 UTC m=+32.688100451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457378 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457409 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457399522 +0000 UTC m=+32.688173305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457419 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457456 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457444694 +0000 UTC m=+32.688218465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : secret "router-metrics-certs-default" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457465 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457474 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fd88cff75-jmrdx: secret "image-registry-tls" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls podName:25c4713f-3616-4d01-b7c7-310f70a14127 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457498813 +0000 UTC m=+32.688272589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls") pod "image-registry-fd88cff75-jmrdx" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127") : secret "image-registry-tls" not found Oct 30 08:53:46.457691 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457683 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:46.457911 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:46.457738 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:48.457722188 +0000 UTC m=+32.688495942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:46.489392 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.489331 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" event={"ID":"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed","Type":"ContainerStarted","Data":"bc15f483dae39cd98863ef82ed45e03d89bf117b92f611bbf12d234145ab65fd"} Oct 30 08:53:46.491775 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.491292 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" event={"ID":"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169","Type":"ContainerStarted","Data":"58e8e17ede929cb60c1eeac7ed01ecf02a1e01039c30b26c15e90189fa4e4dac"} Oct 30 08:53:46.495155 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.494625 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ccr66" event={"ID":"0d9c9c95-575e-4d45-9416-2c8bd643a5e3","Type":"ContainerStarted","Data":"6b2260d44b69670f0d7f3a3a800bc2f3d518cd04746e15a991ec061c6c88415a"} Oct 30 08:53:46.499737 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.499333 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" event={"ID":"627ed97e-e994-4439-815e-d7fe229402e5","Type":"ContainerStarted","Data":"e6b9813fa2ca4cc20ebff0fdc2430c89bd8b0109b8e4ca41638fd092fb2cdc99"} Oct 30 08:53:46.502498 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.502475 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" event={"ID":"8ae3e225-68cf-4814-8d33-e184ad008b3c","Type":"ContainerStarted","Data":"7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14"} Oct 30 08:53:46.531941 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.531892 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" podStartSLOduration=410.531877266 podStartE2EDuration="6m50.531877266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:46.530840095 +0000 UTC m=+30.761613870" watchObservedRunningTime="2026-10-30 08:53:46.531877266 +0000 UTC m=+30.762651041" Oct 30 08:53:46.614672 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.614371 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ccr66" podStartSLOduration=410.614349245 podStartE2EDuration="6m50.614349245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:46.575386936 +0000 UTC m=+30.806160711" watchObservedRunningTime="2026-10-30 08:53:46.614349245 +0000 UTC m=+30.845123024" Oct 30 08:53:46.623269 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.623145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:46.755522 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:46.755367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:47.452612 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:47.452461 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:47.536303 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:47.536256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:48.478805 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.478772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:48.478980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.478819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:48.478980 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.478852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:48.479140 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.479077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:48.479140 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.478952 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:48.479140 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.479132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:48.479298 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.479169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:48.479298 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479198 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.47917426 +0000 UTC m=+36.709948011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:48.479298 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479207 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:48.479298 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.479250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:48.479298 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479271 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.47925556 +0000 UTC m=+36.710029350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.479308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479332 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479320597 +0000 UTC m=+36.710094364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479368 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.478995 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479395 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fd88cff75-jmrdx: secret "image-registry-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479400 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479390873 +0000 UTC m=+36.710164652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479424 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls podName:25c4713f-3616-4d01-b7c7-310f70a14127 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479415063 +0000 UTC m=+36.710188848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls") pod "image-registry-fd88cff75-jmrdx" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127") : secret "image-registry-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479004 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479452 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479466 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65ffb76754-ndfbg: secret "image-registry-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479531 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:48.479550 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479454 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479445984 +0000 UTC m=+36.710219772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : secret "router-metrics-certs-default" not found Oct 30 08:53:48.480081 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479571 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls podName:c86dd71b-65b4-48bc-9b09-772f6dd7053a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479559483 +0000 UTC m=+36.710333237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls") pod "image-registry-65ffb76754-ndfbg" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a") : secret "image-registry-tls" not found Oct 30 08:53:48.480081 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:48.479586 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls podName:c9e051aa-16cd-43b2-aec2-4f19f35ab485 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.479577414 +0000 UTC m=+36.710351167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls") pod "dns-default-mj9w5" (UID: "c9e051aa-16cd-43b2-aec2-4f19f35ab485") : secret "dns-default-metrics-tls" not found Oct 30 08:53:48.515548 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.515496 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" event={"ID":"412a8656-9cb2-4e67-9752-f179cd02219b","Type":"ContainerStarted","Data":"0137b3ef8608d5a529c1a005b997b5c48196bfe4f37bff56cee464226edc39a4"} Oct 30 08:53:48.554335 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.554275 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" podStartSLOduration=380.303496372 podStartE2EDuration="6m22.554255842s" totalImagesPullingTime="2.25075947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:48.554051638 +0000 UTC m=+32.784825412" watchObservedRunningTime="2026-10-30 08:53:48.554255842 +0000 UTC m=+32.785029620" Oct 30 08:53:48.556230 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.555330 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" podStartSLOduration=412.555312771 podStartE2EDuration="6m52.555312771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:46.614653209 +0000 UTC m=+30.845426968" watchObservedRunningTime="2026-10-30 08:53:48.555312771 +0000 UTC m=+32.786086548" Oct 30 08:53:48.644440 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.644402 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:48.744043 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:48.743953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:49.315110 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:49.315029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:49.394122 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:49.394082 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:50.096407 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.096371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:50.096863 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.096427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:50.096863 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:50.096539 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:53:50.096863 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:50.096618 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:22.096599902 +0000 UTC m=+66.327373675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : secret "metrics-daemon-secret" not found Oct 30 08:53:50.103642 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.103612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:50.197781 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.197756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:50.200527 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.200503 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:50.377170 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.377091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:50.386344 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.386312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:50.525005 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.524977 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/0.log" Oct 30 08:53:50.525153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.525023 2577 generic.go:356] "Generic (PLEG): container finished" podID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerID="49230f72e3c65e4410b0c34b6156e9e15246209ca5698a83bdeebb2b3e6e466a" exitCode=255 Oct 30 08:53:50.525153 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.525118 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerDied","Data":"49230f72e3c65e4410b0c34b6156e9e15246209ca5698a83bdeebb2b3e6e466a"} Oct 30 08:53:50.525387 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.525370 2577 scope.go:122] "RemoveContainer" containerID="49230f72e3c65e4410b0c34b6156e9e15246209ca5698a83bdeebb2b3e6e466a" Oct 30 08:53:50.526846 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.526822 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" event={"ID":"47f381ca-b571-4170-a89c-ecdef16c72f6","Type":"ContainerStarted","Data":"a563c6b875057f5e3bf261c6186f378e52117cd7b099a38d87de755dce51b83f"} Oct 30 08:53:50.533877 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.533850 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tn25f" event={"ID":"8f93bd2e-a21e-4bc1-993d-117adc818868","Type":"ContainerStarted","Data":"0d0935d2c6f63c57543fabf1561daa643bcf726e062c80674811e59463e5ceb5"} Oct 30 08:53:50.535507 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.535484 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" event={"ID":"8ae3e225-68cf-4814-8d33-e184ad008b3c","Type":"ContainerStarted","Data":"4a85237789a4bb96357bc56d3295526688f85c0c83a687c92275d942d9ea3771"} Oct 30 08:53:50.606496 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.606440 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" podStartSLOduration=380.899932198 podStartE2EDuration="6m25.606425385s" totalImagesPullingTime="4.706493187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:50.605066994 +0000 UTC m=+34.835840768" watchObservedRunningTime="2026-10-30 08:53:50.606425385 +0000 UTC m=+34.837199155" Oct 30 08:53:50.646839 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.646787 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tn25f" podStartSLOduration=381.946049043 podStartE2EDuration="6m26.646774112s" totalImagesPullingTime="4.700725069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:50.645643966 +0000 UTC m=+34.876417739" watchObservedRunningTime="2026-10-30 08:53:50.646774112 +0000 UTC m=+34.877547886" Oct 30 08:53:50.687499 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.687438 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" podStartSLOduration=382.001968868 podStartE2EDuration="6m26.68741615s" totalImagesPullingTime="4.685447282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:50.686294031 +0000 UTC m=+34.917067806" watchObservedRunningTime="2026-10-30 08:53:50.68741615 +0000 UTC m=+34.918189926" Oct 30 08:53:50.891766 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.890733 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sxh78"] Oct 30 08:53:50.905501 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.903969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:50.914252 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.914122 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxh78"] Oct 30 08:53:50.922524 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.922195 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:53:50.922524 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.922301 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:53:50.922707 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.922593 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:53:50.982103 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:50.982068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.008459 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.008418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbwb\" (UniqueName: \"kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.008459 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.008471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.008706 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.008596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.008706 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.008640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.008805 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.008784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.052398 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.052369 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p9764"] Oct 30 08:53:51.054457 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:51.054430 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7f6695_43a2_4f59_83c5_ee24748be7f5.slice/crio-585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474 WatchSource:0}: Error finding container 585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474: Status 404 returned error can't find the container with id 585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474 Oct 30 08:53:51.064877 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.064856 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xb9sq"] Oct 30 08:53:51.065741 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:51.065720 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00097a4_0c42_452a_af4f_041cf40aecab.slice/crio-5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b WatchSource:0}: Error finding container 5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b: Status 404 returned error can't find the container with id 5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b Oct 30 08:53:51.069617 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.069591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.109847 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.109821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbwb\" (UniqueName: \"kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.109861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.109907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.109935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.110008 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.110024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.110070 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:51.610052287 +0000 UTC m=+35.840826051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : configmap "kube-rbac-proxy" not found Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.110132 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.110196 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.110180 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:51.610165462 +0000 UTC m=+35.840939242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.110587 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.110424 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.110587 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.110500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.130278 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.130250 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbwb\" (UniqueName: \"kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.253461 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.253421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.355738 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.355685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.527982 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.527872 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.539973 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.539943 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:51.540432 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.540415 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/0.log" Oct 30 08:53:51.540553 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.540448 2577 generic.go:356] "Generic (PLEG): container finished" podID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" exitCode=255 Oct 30 08:53:51.540553 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.540479 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerDied","Data":"870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8"} Oct 30 08:53:51.540553 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.540540 2577 scope.go:122] "RemoveContainer" containerID="49230f72e3c65e4410b0c34b6156e9e15246209ca5698a83bdeebb2b3e6e466a" Oct 30 08:53:51.540815 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.540801 2577 scope.go:122] "RemoveContainer" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" Oct 30 08:53:51.541039 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.541016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:53:51.542626 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.542591 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" event={"ID":"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed","Type":"ContainerStarted","Data":"2ff0b1d3956d8cd55a65b8fc6638349cb6d4f498d3c1f7decee20db017e27d21"} Oct 30 08:53:51.544015 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.543964 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p9764" event={"ID":"7d7f6695-43a2-4f59-83c5-ee24748be7f5","Type":"ContainerStarted","Data":"585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474"} Oct 30 08:53:51.545545 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.545524 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xb9sq" event={"ID":"b00097a4-0c42-452a-af4f-041cf40aecab","Type":"ContainerStarted","Data":"e98df0b8e88d5ff7df2226d3056e5d80ad80c37e219eb9af052df900bb446380"} Oct 30 08:53:51.545632 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.545560 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xb9sq" event={"ID":"b00097a4-0c42-452a-af4f-041cf40aecab","Type":"ContainerStarted","Data":"5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b"} Oct 30 08:53:51.545632 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.545583 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:51.546944 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.546917 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbc5k" event={"ID":"11e00d62-c126-48fb-aab2-d1eac679b710","Type":"ContainerStarted","Data":"94770b5bc0d6f11dcd2e862d875a78fe43040b542dc2dfa90f8f427008470422"} Oct 30 08:53:51.617722 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.617629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.617952 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.617815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:51.618079 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.617986 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.618176 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.618160 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.618138723 +0000 UTC m=+36.848912501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.619109 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.618555 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:51.619109 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:51.618625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.618610253 +0000 UTC m=+36.849384020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : configmap "kube-rbac-proxy" not found Oct 30 08:53:51.621880 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.621850 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x"] Oct 30 08:53:51.655403 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.655372 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x"] Oct 30 08:53:51.655569 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.655480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:51.661392 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.661348 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" podStartSLOduration=373.36620526 podStartE2EDuration="6m18.661333223s" totalImagesPullingTime="5.295127963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:51.66046984 +0000 UTC m=+35.891243619" watchObservedRunningTime="2026-10-30 08:53:51.661333223 +0000 UTC m=+35.892106997" Oct 30 08:53:51.677527 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.677493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.706603 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.706545 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vbc5k" podStartSLOduration=2.4147686999999998 podStartE2EDuration="7.706530877s" totalImagesPullingTime="5.291762177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:51.706056478 +0000 UTC m=+35.936830252" watchObservedRunningTime="2026-10-30 08:53:51.706530877 +0000 UTC m=+35.937304652" Oct 30 08:53:51.722456 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.722428 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h5xkm\"" Oct 30 08:53:51.745877 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.745845 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 08:53:51.745877 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.745843 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:51.747232 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.747161 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xb9sq" podStartSLOduration=35.747146252 podStartE2EDuration="35.747146252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:51.7460026 +0000 UTC m=+35.976776374" watchObservedRunningTime="2026-10-30 08:53:51.747146252 +0000 UTC m=+35.977920023" Oct 30 08:53:51.763699 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.763628 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.819828 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.819792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr58\" (UniqueName: \"kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58\") pod \"migrator-66df5d496b-4595x\" (UID: \"29d6f07b-b204-4274-9d60-f6f7a32b4a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:51.863492 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.863452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.921367 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.921328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr58\" (UniqueName: \"kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58\") pod \"migrator-66df5d496b-4595x\" (UID: \"29d6f07b-b204-4274-9d60-f6f7a32b4a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:51.942047 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.942020 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr58\" (UniqueName: \"kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58\") pod \"migrator-66df5d496b-4595x\" (UID: \"29d6f07b-b204-4274-9d60-f6f7a32b4a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:51.964880 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.964848 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:51.976872 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:51.976850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:52.280057 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.280025 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x"] Oct 30 08:53:52.281017 ip-10-0-129-105 kubenswrapper[2577]: W1030 08:53:52.280991 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d6f07b_b204_4274_9d60_f6f7a32b4a90.slice/crio-0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f WatchSource:0}: Error finding container 0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f: Status 404 returned error can't find the container with id 0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f Oct 30 08:53:52.376379 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.376349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:52.526307 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:52.526307 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526442 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526525 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fd88cff75-jmrdx: secret "image-registry-tls" not found Oct 30 08:53:52.526549 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526492 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526562 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526548078 +0000 UTC m=+44.757321830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526497 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526589 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65ffb76754-ndfbg: secret "image-registry-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526596 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526611 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls podName:25c4713f-3616-4d01-b7c7-310f70a14127 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526600543 +0000 UTC m=+44.757374303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls") pod "image-registry-fd88cff75-jmrdx" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127") : secret "image-registry-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526502 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526627 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526618979 +0000 UTC m=+44.757392734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.526592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526627 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526650 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls podName:c86dd71b-65b4-48bc-9b09-772f6dd7053a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526642153 +0000 UTC m=+44.757415911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls") pod "image-registry-65ffb76754-ndfbg" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a") : secret "image-registry-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526668 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526664276 +0000 UTC m=+44.757438035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526690 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls podName:c9e051aa-16cd-43b2-aec2-4f19f35ab485 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.52668319 +0000 UTC m=+44.757456947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls") pod "dns-default-mj9w5" (UID: "c9e051aa-16cd-43b2-aec2-4f19f35ab485") : secret "dns-default-metrics-tls" not found Oct 30 08:53:52.526832 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526733 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526698323 +0000 UTC m=+44.757472080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : secret "router-metrics-certs-default" not found Oct 30 08:53:52.527435 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.526752 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:54:00.526744405 +0000 UTC m=+44.757518157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:52.552504 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.552482 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:52.552880 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.552850 2577 scope.go:122] "RemoveContainer" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" Oct 30 08:53:52.553081 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.553059 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:53:52.553766 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.553736 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" event={"ID":"29d6f07b-b204-4274-9d60-f6f7a32b4a90","Type":"ContainerStarted","Data":"0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f"} Oct 30 08:53:52.627773 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.627732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:52.627950 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:52.627881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:52.627950 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.627889 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:52.627950 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.627948 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:54.627930874 +0000 UTC m=+38.858704630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:52.628125 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.627996 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:52.628125 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:52.628028 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:54.628020052 +0000 UTC m=+38.858793818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : configmap "kube-rbac-proxy" not found Oct 30 08:53:54.565227 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:54.565187 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" event={"ID":"29d6f07b-b204-4274-9d60-f6f7a32b4a90","Type":"ContainerStarted","Data":"a1de0a3872f59bd582014ce92b6a58d7cfd7c0d880f226d742f630349f118b1f"} Oct 30 08:53:54.604596 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:54.604542 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" podStartSLOduration=2.077857931 podStartE2EDuration="3.604528582s" totalImagesPullingTime="1.526670651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:54.604189092 +0000 UTC m=+38.834962891" watchObservedRunningTime="2026-10-30 08:53:54.604528582 +0000 UTC m=+38.835302356" Oct 30 08:53:54.649173 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:54.649133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:54.649350 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:54.649324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:54.649433 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:54.649390 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:54.649489 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:54.649431 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:54.649489 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:54.649471 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.649451075 +0000 UTC m=+42.880224844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : configmap "kube-rbac-proxy" not found Oct 30 08:53:54.649589 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:54.649492 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.649482489 +0000 UTC m=+42.880256246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:55.052728 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:55.052678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:55.058559 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:55.058534 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:55.058697 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:55.058592 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:55.058965 ip-10-0-129-105 kubenswrapper[2577]: I1030 08:53:55.058949 2577 scope.go:122] "RemoveContainer" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" Oct 30 08:53:55.059172 ip-10-0-129-105 kubenswrapper[2577]: E1030 08:53:55.059151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:53:56.192605 ip-10-0-129-105 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 08:53:56.203654 ip-10-0-129-105 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 08:53:56.203826 ip-10-0-129-105 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 08:53:56.204002 ip-10-0-129-105 systemd[1]: kubelet.service: Consumed 3.071s CPU time. Oct 30 08:53:56.213945 ip-10-0-129-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:53:56.305965 ip-10-0-129-105 kubenswrapper[4920]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:56.305965 ip-10-0-129-105 kubenswrapper[4920]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:53:56.305965 ip-10-0-129-105 kubenswrapper[4920]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:56.305965 ip-10-0-129-105 kubenswrapper[4920]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:56.308328 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308239 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.308328 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308327 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.308328 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308330 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308335 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308339 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308342 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308345 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308348 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308351 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308354 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308357 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308360 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308363 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308365 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308368 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308371 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308374 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308377 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308379 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308382 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308385 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308387 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.308432 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308390 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308393 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308395 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308398 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308401 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308412 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308416 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308419 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308421 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308424 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308427 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308430 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308433 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308435 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308438 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308441 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308444 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308447 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308451 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308453 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.308933 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308456 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308459 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308462 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308464 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308467 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308472 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308476 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308480 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308482 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308485 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308488 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308491 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308494 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308497 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308500 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308502 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308505 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308508 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308511 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308514 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.309445 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308516 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308519 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308522 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308525 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308528 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308531 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308534 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308536 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308539 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308541 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308544 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308546 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308549 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308552 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308562 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308565 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308568 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308570 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308573 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.309925 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308576 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308579 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308582 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308585 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308587 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308590 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308593 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308596 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308599 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308601 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308604 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308607 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308610 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308612 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308615 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308618 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308620 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308623 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308626 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.310400 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308629 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308634 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308638 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308641 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308644 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.308647 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309525 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309534 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309538 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309543 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309546 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309549 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309552 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309556 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309558 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309562 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309565 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309567 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309570 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.310872 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309573 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309576 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309578 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309581 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309584 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309588 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309590 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309593 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309596 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309598 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309601 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309604 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309606 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309609 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309613 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309617 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309620 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309623 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309626 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.311365 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309628 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309631 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309633 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309636 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309640 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309643 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309645 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309648 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309650 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309653 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309656 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309659 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309663 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309667 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309670 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309673 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309675 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309678 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309681 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.311834 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309684 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309686 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309689 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309692 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309694 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309697 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309700 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309702 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309705 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309708 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309710 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309713 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309715 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309719 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309722 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309725 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309728 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309730 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309734 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309736 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.312312 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309739 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309741 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309745 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309747 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309750 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309752 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309755 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309757 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309760 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309763 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309766 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309769 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309771 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309774 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309776 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309779 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309781 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309784 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309786 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309789 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.312836 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309792 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309794 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309797 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309799 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309801 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309804 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309807 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309810 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309812 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309815 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309818 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309821 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309825 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309827 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.309830 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310362 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310369 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310373 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310377 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.313350 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310380 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310382 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310386 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310389 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310392 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310395 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310397 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310400 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310403 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310406 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310409 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310412 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310414 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310417 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310420 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310422 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310425 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310428 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310431 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.313812 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310433 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310436 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310439 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310442 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310444 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310452 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310455 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310457 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310460 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310462 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310465 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310467 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310470 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310472 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310475 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310478 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310484 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310488 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310491 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.314288 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310494 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310498 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310502 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310506 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310509 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310512 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310515 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310518 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310521 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310524 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310527 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310531 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310534 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310537 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310540 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310543 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310546 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310549 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310552 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.314768 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310555 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310558 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310564 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310567 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310569 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310572 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310574 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310577 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310580 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310582 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310586 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310589 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310592 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310596 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310600 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310604 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310608 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310612 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310615 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310618 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.315240 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310621 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310623 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310626 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310628 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310639 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310642 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310645 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310647 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310650 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310653 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310656 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310658 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310661 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310664 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310669 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310673 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310675 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310678 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310681 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310685 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.315736 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310688 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310690 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310694 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310697 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.310699 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310786 4920 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310793 4920 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310799 4920 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310804 4920 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310809 4920 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310812 4920 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310817 4920 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310821 4920 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310824 4920 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310827 4920 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310831 4920 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310835 4920 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310838 4920 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310841 4920 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310844 4920 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310847 4920 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310850 4920 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310854 4920 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:53:56.316233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310858 4920 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310862 4920 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310865 4920 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310871 4920 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310875 4920 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310879 4920 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310882 4920 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310886 4920 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310890 4920 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310893 4920 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310896 4920 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310900 4920 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310904 4920 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310907 4920 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310911 4920 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310914 4920 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310917 4920 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310920 4920 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310924 4920 flags.go:64] FLAG: --enable-server="true" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310927 4920 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310931 4920 flags.go:64] FLAG: --event-burst="100" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310934 4920 flags.go:64] FLAG: --event-qps="50" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310937 4920 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310940 4920 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:53:56.316799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310943 4920 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310948 4920 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310951 4920 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310954 4920 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310957 4920 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310960 4920 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310963 4920 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310966 4920 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310970 4920 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310975 4920 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310979 4920 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310984 4920 flags.go:64] FLAG: --feature-gates="" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310989 4920 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310992 4920 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310996 4920 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.310999 4920 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311002 4920 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311006 4920 flags.go:64] FLAG: --help="false" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311009 4920 flags.go:64] FLAG: --hostname-override="ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311014 4920 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311017 4920 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311020 4920 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311023 4920 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:53:56.317413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311027 4920 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311030 4920 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311033 4920 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311036 4920 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311039 4920 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311042 4920 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311045 4920 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311048 4920 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311051 4920 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311054 4920 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311057 4920 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311060 4920 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311063 4920 flags.go:64] FLAG: --lock-file="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311066 4920 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311070 4920 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311075 4920 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311081 4920 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311084 4920 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311088 4920 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311091 4920 flags.go:64] FLAG: --logging-format="text" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311096 4920 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311099 4920 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311103 4920 flags.go:64] FLAG: --manifest-url="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311106 4920 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311110 4920 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:53:56.317999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311114 4920 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311118 4920 flags.go:64] FLAG: --max-pods="110" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311121 4920 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311127 4920 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311130 4920 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311134 4920 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311136 4920 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311139 4920 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311142 4920 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311150 4920 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311153 4920 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311156 4920 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311159 4920 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311162 4920 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311166 4920 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311169 4920 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311172 4920 flags.go:64] FLAG: --port="10250" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311175 4920 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311178 4920 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7ca2e4dd64f4aae" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311181 4920 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311184 4920 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311187 4920 flags.go:64] FLAG: --register-node="true" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311190 4920 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311194 4920 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311197 4920 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311200 4920 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:53:56.318675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311203 4920 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311208 4920 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311211 4920 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311229 4920 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311233 4920 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311236 4920 flags.go:64] FLAG: --runonce="false" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311239 4920 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311242 4920 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311245 4920 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311250 4920 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311253 4920 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311256 4920 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311259 4920 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311263 4920 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311266 4920 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311269 4920 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311272 4920 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311275 4920 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311278 4920 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311281 4920 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311284 4920 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311289 4920 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311293 4920 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311296 4920 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311300 4920 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:53:56.319637 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311303 4920 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311306 4920 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311309 4920 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311312 4920 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311315 4920 flags.go:64] FLAG: --v="2" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311319 4920 flags.go:64] FLAG: --version="false" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311324 4920 flags.go:64] FLAG: --vmodule="" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311327 4920 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311332 4920 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311482 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311486 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311489 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311493 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311496 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311499 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311502 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311506 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311508 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311511 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311514 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311516 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311519 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.320318 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311522 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311525 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311528 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311531 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311534 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311536 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311539 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311542 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311544 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311547 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311550 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311552 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311555 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311558 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311561 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311563 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311566 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311568 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311573 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311576 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.320991 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311578 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311581 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311584 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311587 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311590 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311592 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311595 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311598 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311600 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311603 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311606 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311608 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311611 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311614 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311618 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311622 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311625 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311628 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311631 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.321791 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311634 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311636 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311639 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311642 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311645 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311647 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311650 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311653 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311655 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311658 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311660 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311664 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311668 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311673 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311676 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311678 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311681 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311684 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311687 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.322380 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311690 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311693 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311695 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311699 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311701 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311704 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311706 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311709 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311712 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311715 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311717 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311720 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311723 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311725 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311728 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311731 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311734 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311736 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311739 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311742 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.322851 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311744 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311747 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311750 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311752 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311757 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311760 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311763 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311765 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311768 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311771 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311773 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311776 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311779 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311782 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311784 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.311809 4920 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:56.323622 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311848 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311853 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311857 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311860 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311863 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311866 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311869 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311873 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311877 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311880 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311882 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311885 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311888 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311891 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311893 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311896 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311899 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311901 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311904 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.324128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311907 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311911 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311914 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311917 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311920 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311922 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311925 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311928 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311931 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311933 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311937 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311939 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311943 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311945 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311948 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311950 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311953 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311956 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311958 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311961 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.324695 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311964 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311966 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311969 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311971 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311974 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311976 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311979 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311982 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311984 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311987 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311990 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311992 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311995 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.311999 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312003 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312005 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312008 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312011 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312013 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312016 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.325176 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312018 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312021 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312024 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312027 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312030 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312033 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312035 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312038 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312041 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312043 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312046 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312048 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312051 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312054 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312056 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312059 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312061 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312064 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312067 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312070 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.325685 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312072 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312075 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312077 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312080 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312083 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312087 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312089 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312092 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312094 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312097 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312100 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312102 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312105 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312108 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312110 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312114 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312118 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312122 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312125 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.326195 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312128 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312131 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312134 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312136 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312139 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312142 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312144 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.312147 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.316135 4920 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.316152 4920 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.316172 4920 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.316177 4920 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316262 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316268 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316272 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316275 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316278 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316281 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316283 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.326820 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316287 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316290 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316292 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316295 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316298 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316301 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316303 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316307 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316309 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316319 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316322 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316325 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316328 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316330 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316337 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316342 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316346 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316349 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316352 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.327337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316356 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316359 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316362 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316365 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316368 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316370 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316373 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316376 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316378 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316381 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316384 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316386 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316389 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316391 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316394 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316396 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316399 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316401 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316404 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.327794 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316407 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316409 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316412 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316415 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316424 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316427 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316430 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316433 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316436 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316438 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316441 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316444 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316446 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316449 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316451 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316454 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316456 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316459 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316462 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316465 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.328397 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316467 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316470 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316472 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316475 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316478 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316480 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316483 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316487 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316491 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316494 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316497 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316500 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316502 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316505 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316507 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316511 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316514 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316522 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316525 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.328930 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316528 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316531 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316533 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316536 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316538 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316541 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316544 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316546 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316549 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316552 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316554 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316557 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316559 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316562 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316564 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316567 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316570 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316573 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316575 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316578 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.329408 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316581 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316584 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.316612 4920 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316804 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316809 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316813 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316817 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316821 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316824 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316827 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316830 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316839 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316842 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316844 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316847 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.329883 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316850 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316852 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316855 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316858 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316860 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316863 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316865 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316868 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316870 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316873 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316876 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316878 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316881 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316883 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316886 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316891 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316894 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316897 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316899 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.330249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316902 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316905 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316908 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316910 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316913 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316915 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316918 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316921 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316923 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316926 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316934 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316937 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316943 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316946 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316948 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316951 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316953 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316956 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316959 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316962 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.330711 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316964 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316967 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316970 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316973 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316975 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316978 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316980 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316983 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316987 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316990 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316993 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316996 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.316998 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317001 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317004 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317006 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317009 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317012 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317014 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.331193 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317017 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317019 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317022 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317025 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317033 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317036 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317039 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317041 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317044 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317046 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317049 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317053 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317057 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317060 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317063 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317065 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317068 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317071 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317073 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.331665 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317076 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317079 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317083 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317085 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317088 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317090 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317093 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317096 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317099 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317101 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317104 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317107 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317110 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317112 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317115 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317118 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.332337 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317120 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.317143 4920 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317200 4920 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317204 4920 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317208 4920 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317211 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317233 4920 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317237 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317240 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317243 4920 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317247 4920 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317252 4920 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317255 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317259 4920 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317261 4920 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317264 4920 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:56.332910 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317267 4920 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317269 4920 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317272 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317276 4920 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317279 4920 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317283 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317285 4920 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317288 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317291 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317293 4920 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317296 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317299 4920 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317302 4920 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317304 4920 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317307 4920 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317310 4920 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317312 4920 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317315 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317318 4920 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317327 4920 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:56.333426 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317330 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317332 4920 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317336 4920 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317340 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317343 4920 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317346 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317348 4920 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317352 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317354 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317357 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317359 4920 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317362 4920 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317364 4920 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317367 4920 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317369 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317373 4920 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317376 4920 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317380 4920 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317383 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:56.334168 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317386 4920 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317388 4920 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317391 4920 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317393 4920 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317396 4920 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317399 4920 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317402 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317404 4920 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317407 4920 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317410 4920 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317412 4920 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317415 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317418 4920 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317427 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317430 4920 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317432 4920 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317435 4920 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317438 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317440 4920 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317443 4920 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:56.334913 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317446 4920 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317448 4920 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317451 4920 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317454 4920 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317456 4920 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317459 4920 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317461 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317464 4920 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317468 4920 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317470 4920 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317473 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317476 4920 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317479 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317481 4920 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317484 4920 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317487 4920 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317489 4920 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317492 4920 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317494 4920 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317497 4920 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:56.335691 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317500 4920 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317502 4920 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317505 4920 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317507 4920 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317510 4920 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317513 4920 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317522 4920 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317525 4920 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317527 4920 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317530 4920 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317532 4920 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317535 4920 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:56.317537 4920 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.317590 4920 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.318718 4920 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.318774 4920 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.319010 4920 server.go:1014] "Starting client certificate rotation" Oct 30 08:53:56.336247 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.319171 4920 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.319269 4920 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:16 +0000 UTC" deadline="2028-06-30 23:57:03.283309382 +0000 UTC" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.319342 4920 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14631h3m6.963970386s" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.319653 4920 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.322553 4920 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.324162 4920 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.325776 4920 log.go:25] "Validated CRI v1 image API" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.326492 4920 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:53:56.336657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.328355 4920 fs.go:135] Filesystem UUIDs: map[289d09e1-363d-4a8d-b851-a5d8afd10014:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd7b0745-cba1-4abc-bdc5-0a8f1c0c5b7a:/dev/nvme0n1p4] Oct 30 08:53:56.336910 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.328374 4920 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c5915954fd7b6e75d6d90931cc3576246b848260928f36183513c19ff19dceb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c5915954fd7b6e75d6d90931cc3576246b848260928f36183513c19ff19dceb/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8130978ab436abea23c76b7105e5242296011d95be0797c43e231396d1909880/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8130978ab436abea23c76b7105e5242296011d95be0797c43e231396d1909880/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81aa7b94185e8fee5ca23b9951b10815e719e40dbb678f9864b4a7b7c1a3fb75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81aa7b94185e8fee5ca23b9951b10815e719e40dbb678f9864b4a7b7c1a3fb75/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc15f483dae39cd98863ef82ed45e03d89bf117b92f611bbf12d234145ab65fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc15f483dae39cd98863ef82ed45e03d89bf117b92f611bbf12d234145ab65fd/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2dba5842d55c7692b989399bb119b6f5eaec878271dcbdf3db36f7b3b34b203/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2dba5842d55c7692b989399bb119b6f5eaec878271dcbdf3db36f7b3b34b203/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a50c02c-9976-4034-8785-68c3e971060a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0a50c02c-9976-4034-8785-68c3e971060a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d9c9c95-575e-4d45-9416-2c8bd643a5e3/volumes/kubernetes.io~projected/kube-api-access-hvxwg:{mountpoint:/var/lib/kubelet/pods/0d9c9c95-575e-4d45-9416-2c8bd643a5e3/volumes/kubernetes.io~projected/kube-api-access-hvxwg major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ff1f68b-203c-43a3-b5a5-671eda399b32/volumes/kubernetes.io~projected/kube-api-access-cwcc7:{mountpoint:/var/lib/kubelet/pods/0ff1f68b-203c-43a3-b5a5-671eda399b32/volumes/kubernetes.io~projected/kube-api-access-cwcc7 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~projected/kube-api-access-pf8xk:{mountpoint:/var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~projected/kube-api-access-pf8xk major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/kube-api-access-d6ljr:{mountpoint:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/kube-api-access-d6ljr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29d6f07b-b204-4274-9d60-f6f7a32b4a90/volumes/kubernetes.io~projected/kube-api-access-gfr58:{mountpoint:/var/lib/kubelet/pods/29d6f07b-b204-4274-9d60-f6f7a32b4a90/volumes/kubernetes.io~projected/kube-api-access-gfr58 major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d50e209-0bda-4891-be9d-9e3580efab4e/volumes/kubernetes.io~projected/kube-api-access-898q4:{mountpoint:/var/lib/kubelet/pods/2d50e209-0bda-4891-be9d-9e3580efab4e/volumes/kubernetes.io~projected/kube-api-access-898q4 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/412a8656-9cb2-4e67-9752-f179cd02219b/volumes/kubernetes.io~projected/kube-api-access-fsh6f:{mountpoint:/var/lib/kubelet/pods/412a8656-9cb2-4e67-9752-f179cd02219b/volumes/kubernetes.io~projected/kube-api-access-fsh6f major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~projected/kube-api-access-qw65l:{mountpoint:/var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~projected/kube-api-access-qw65l major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed/volumes/kubernetes.io~projected/kube-api-access-5zt7w:{mountpoint:/var/lib/kubelet/pods/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed/volumes/kubernetes.io~projected/kube-api-access-5zt7w major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b/volumes/kubernetes.io~projected/kube-api-access-brbwb:{mountpoint:/var/lib/kubelet/pods/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b/volumes/kubernetes.io~projected/kube-api-access-brbwb major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61e96f32-b0b7-40b7-97e5-61ff980cc465/volumes/kubernetes.io~projected/kube-api-access-xq6qg:{mountpoint:/var/lib/kubelet/pods/61e96f32-b0b7-40b7-97e5-61ff980cc465/volumes/kubernetes.io~projected/kube-api-access-xq6qg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/627ed97e-e994-4439-815e-d7fe229402e5/volumes/kubernetes.io~projected/kube-api-access-z8h26:{mountpoint:/var/lib/kubelet/pods/627ed97e-e994-4439-815e-d7fe229402e5/volumes/kubernetes.io~projected/kube-api-access-z8h26 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~projected/kube-api-access-bkmts:{mountpoint:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~projected/kube-api-access-bkmts major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d7f6695-43a2-4f59-83c5-ee24748be7f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7d7f6695-43a2-4f59-83c5-ee24748be7f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49/volumes/kubernetes.io~projected/kube-api-access-pcwwt:{mountpoint:/var/lib/kubelet/pods/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49/volumes/kubernetes.io~projected/kube-api-access-pcwwt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~projected/kube-api-access-xc6q6:{mountpoint:/var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~projected/kube-api-access-xc6q6 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d2f4c5d-a352-4c80-8794-edf020ddbafa/volumes/kubernetes.io~projected/kube-api-access-44spc:{mountpoint:/var/lib/kubelet/pods/8d2f4c5d-a352-4c80-8794-edf020ddbafa/volumes/kubernetes.io~projected/kube-api-access-44spc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f7411dc-7fed-43b4-be33-7eca7c3144d5/volumes/kubernetes.io~projected/kube-api-access-hwwx2:{mountpoint:/var/lib/kubelet/pods/8f7411dc-7fed-43b4-be33-7eca7c3144d5/volumes/kubernetes.io~projected/kube-api-access-hwwx2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~projected/kube-api-access-9fkth:{mountpoint:/var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~projected/kube-api-access-9fkth major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~projected/kube-api-access-v9562:{mountpoint:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~projected/kube-api-access-v9562 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a26a8219-f82f-408c-9241-a82eec0b312d/volumes/kubernetes.io~projected/kube-api-access-2sswm:{mountpoint:/var/lib/kubelet/pods/a26a8219-f82f-408c-9241-a82eec0b312d/volumes/kubernetes.io~projected/kube-api-access-2sswm major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~projected/kube-api-access-5r7r7:{mountpoint:/var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~projected/kube-api-access-5r7r7 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b00097a4-0c42-452a-af4f-041cf40aecab/volumes/kubernetes.io~projected/kube-api-access-9vf5k:{mountpoint:/var/lib/kubelet/pods/b00097a4-0c42-452a-af4f-041cf40aecab/volumes/kubernetes.io~projected/kube-api-access-9vf5k major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/kube-api-access-4pm8d:{mountpoint:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/kube-api-access-4pm8d major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9e051aa-16cd-43b2-aec2-4f19f35ab485/volumes/kubernetes.io~projected/kube-api-access-pnhcb:{mountpoint:/var/lib/kubelet/pods/c9e051aa-16cd-43b2-aec2-4f19f35ab485/volumes/kubernetes.io~projected/kube-api-access-pnhcb major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0a1416c-5f6d-4ac7-9f38-2760a05daa83/volumes/kubernetes.io~projected/kube-api-access-whjdg:{mountpoint:/var/lib/kubelet/pods/d0a1416c-5f6d-4ac7-9f38-2760a05daa83/volumes/kubernetes.io~projected/kube-api-access-whjdg major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~projected/kube-api-access-h44jz:{mountpoint:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~projected/kube-api-access-h44jz major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb312004-8950-4bea-ab8c-2c5279052edf/volumes/kubernetes.io~projected/kube-api-access-bmnnc:{mountpoint:/var/lib/kubelet/pods/fb312004-8950-4bea-ab8c-2c5279052edf/volumes/kubernetes.io~projected/kube-api-access-bmnnc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169/volumes/kubernetes.io~projected/kube-api-access-mwk7n:{mountpoint:/var/lib/kubelet/pods/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169/volumes/kubernetes.io~projected/kube-api-access-mwk7n major:0 minor:226 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bc460a9b89dd6dc8007dab9703ba978df9742df35e58a64d6c96f8f677206a0b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5fdd1c04a733f12f2b4b8d29774594c66743f3a3e8c558c9a8b4a468ae97e40e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2b1fb84c72615ee2bb488871c6c7238edbbb510a9fb91aa6af794ed4f9dbf443/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5133ea28d252dbd8c089840e357a16101b88c3e531dcf12c0a4c9e881d2428c2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3d80597318e4fb562916363e7f4b9e5b710749287c4f85d2cb45e5798f0ed2a0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b41ae721908a3db64646e18359ba7741adc387cf523ef949049ac4499f8eeeb8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/667536aba4c302f25586dab96e68fc887e66c4f44ba6af287564bea1c1012023/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e9510cde4d6d66cb3290ebbd3c1987cab85534add593195e3fffd62b4f651180/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f41e2f7efcefbf1ba793b576fd8a91997ae1d8665e60e26e5fbf211eb6c6942c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/347ff6013cce0712b4e36f954840d23dd5003f0d55ae0fdddd0721956f5b47ac/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/1671dfe56d9c4dbfc3910d72ca2de90a9c8363a4cd26e19aeedbc7ce9b1fb0a2/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/98e529d84c125f990165ac4d53f14dc8f8c3806e20ef3915a15aaee7fade7996/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4f222c0141659f2745566356bfac133b553dd52ad8a74ebc38a1bde334ab9766/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4b991b6fb292dedad7d0d112b3dc9485e9935c45defeb4c0870c40f55be78787/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/908c45f64adabe5aac37a338008606a51631def3fb8bbae1eb858bd7073adcf1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0c72f12c9d8df38862ff0e0fe00733f6e7a05c81dc4b3353d728ac0741722811/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/94733a2823bca68d112733b1b4dd2e02140c6df15d7e8b6cb1a95092e0fc1723/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/859a49d7fddf6cb5fae50917040d954f751759b04512b5c01931bcb7b3ae267b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fbb5703cde9a082e76dfbb95a5e723b88810ea0998cc15bf5b7f396b5ac69850/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1b74b0362588c45f1c81c7a2e9e43750494e372c734921beed629bd1b2d99417/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b60e7ca7a9a994d96f5b80e0857d3f969d8fbdb906709f0c7a5fb22b03926ea2/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/3d19582abfbadb7e6d9fc7ef7d5ac4e897239c8d9f9674dcb018e149ab2ab22d/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c9005f18a445266995ac27b2e12cc0d1372665af6fdc5d97ca6b505e2cd14ece/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8609539f92ba55021e71862ed6a329d442dfd101d838c20fc411cb16665d407b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/cd324a515141d47dfd6ab905a79eb8b99cc7a48a35303ca0230731d5e46ef140/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/dbf1e5729bd6176b486406e8ad0634ea5fff51473bc5b9bf7133868d65697019/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a0e5743839613fdfef418338e7ae0712280c069a93396eb455f4667fcb5c417a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b45a6a1051b596af6a135247d52a8c8bc3ae8963079cec7a488e25576488a731/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/531116a818610c7e57d50d00ae18464253ea1f816735c60b6d63a314766465b7/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/c2b6e29e0bfcf61fd569788db14a761fb9bee7fb02701bfb4f92e18b713096c3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3cdb7c8f361de586b397f0fba32b5cc60983112f2ab3c658f48c28b302186f4e/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/04cae75a6a5a55efbeb375abc71614d3170295a350c3c407199b7c699a5b23fc/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/55c1a6d1fa77df2619e56ea09d3fa090f9d2abee4b6881c0a73b6e504dfe8c4b/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/944cb0d3a67c2e09447cfa7781e1f36ad2c1be1d26eb8624caa0b8c82cd4c542/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/9ad5b3d215467fe70a9db1ff14c2610a0599b5e73c7174c05c6f0c69b57a5ff3/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/4e1647a3574c85365d80699cd7395a0df8fc36810f646ab0cf9d8c9e75c709b4/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/20d35b185c725c4ffd122a876190bc56c1ff66d480e1a8a7929688aca0688dde/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/44fcdb5bb9e9993849ae1bc5b5379adf7ab624dc54ceaecea22d68669c4e5c22/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/47e6d24140417d93a330e707570de9a56054ca5e8a485256b2eab3771137634e/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/a9e93ca9c72424aea78e358e1c3ea33cbfd0160a152d0baedea6b0267b4ff5b6/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/03e31a0a40cfe45a4fde5d2fe6a7760d2db9b633ff134bf6fbd1d23209d1232b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/7a3d973f186ab93f6477eda245f7ddb0a95011d6d0c7087f784b7c09ff551e61/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/400e4f9838108276d96251010dd17ca25eff493354c922f9ce2b39eb857af8ba/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/ad06e90b45d3ea02332b1d51e9d286c974f991d94b18499dc2bdbfdb98048346/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/40b9f2595f0a62908e967b735433f87dace8301c437e0ee6d9cc0a7ee2c10b3b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ceb643291b06d6aa2b7202a2bb4389d0f3b8289b44c89ff54fa648dd1c987ea0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e17296069928f1944ffbeff29439f795a91e5067ae047b249627b32e21275f3c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/467c32486d5b2a90f7e5846e24d74a3f93856c45ccdea19f46b209aa6bb065d6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b94074e655147a7816a669433291529de43bfbbb0f4a706c1743d329134b9000/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/db03c6db9b26f0cab95366fce1e52fcdb032ceaa28694bb3bb99fa55e069788f/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3b60e95200bd9501bda036310e2898997742fc89d5cee845bfb69129a9f2cc39/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fed2206758e243b8917b1af077cca611ca029a07f21b63be974703d3d0194965/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b865e65d0108e79e41f1251585548835aee73a2a51f785379405ea424f04c586/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/ca51b8b25c745b1daee69402d182db940f7bdbf161fd336ac42a46e617c348fe/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/30b9632934f5d076a86988a13ba2cc4e73281aa5ce0e4a2c20031378babcd030/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7498a52c1b9cd25fd846bfe54ea79bffeda3d4dc1e38f8eb6107e50f96970614/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.336195 4920 manager.go:217] Machine: {Timestamp:2026-10-30 08:53:56.335718317 +0000 UTC m=+0.059145451 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128785 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e3bc00e65b2a8a7adee1566c8945 SystemUUID:ec23e3bc-00e6-5b2a-8a7a-dee1566c8945 BootID:35721d21-7570-460b-8399-2673731b2c9b Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~projected/kube-api-access-qw65l DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/627ed97e-e994-4439-815e-d7fe229402e5/volumes/kubernetes.io~projected/kube-api-access-z8h26 DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a26a8219-f82f-408c-9241-a82eec0b312d/volumes/kubernetes.io~projected/kube-api-access-2sswm DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf3aae969dbefccbc3e80e58cd6c30268f45b3cedd436b9121519a4e8aa84083/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d0a1416c-5f6d-4ac7-9f38-2760a05daa83/volumes/kubernetes.io~projected/kube-api-access-whjdg DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc15f483dae39cd98863ef82ed45e03d89bf117b92f611bbf12d234145ab65fd/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0a50c02c-9976-4034-8785-68c3e971060a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8130978ab436abea23c76b7105e5242296011d95be0797c43e231396d1909880/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/969f05b31414431ebbe35120d787b0881c58e1169fbece29fd2a0acba7ab7d9e/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7eb18f72c4a30e96b9b51bff593625ce74389f9a7c059492a19487936634ac14/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b/volumes/kubernetes.io~projected/kube-api-access-brbwb DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47f381ca-b571-4170-a89c-ecdef16c72f6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1fe6e0ed92775ab0551265259dd7eb5f1ef845b1847f3310d4ba424d4486a43/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed/volumes/kubernetes.io~projected/kube-api-access-5zt7w DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/kube-api-access-d6ljr DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169/volumes/kubernetes.io~projected/kube-api-access-mwk7n DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5926140d68db041c47c22d9e81d0525e4cdae0da0568ca65fd969c9b580bdbb/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c2b67459c06c5c42e3ce0c761c9a1e9421afc42d2ddb831524af8c209e7bc0b/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/585c65551856ee6e69abb9a156851665968ed37f7294c5d3b1b69d96b197a474/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fb312004-8950-4bea-ab8c-2c5279052edf/volumes/kubernetes.io~projected/kube-api-access-bmnnc DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e00d62-c126-48fb-aab2-d1eac679b710/volumes/kubernetes.io~projected/kube-api-access-pf8xk DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~projected/kube-api-access-xc6q6 DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adbbc457-38a1-45fd-ab3d-738f7dd07913/volumes/kubernetes.io~projected/kube-api-access-5r7r7 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c5915954fd7b6e75d6d90931cc3576246b848260928f36183513c19ff19dceb/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2dba5842d55c7692b989399bb119b6f5eaec878271dcbdf3db36f7b3b34b203/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bb599c421e7859e9567d92552e8c61985c4b990d65e7af143ddbedbdd299bf4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b22147557ea97e23a54d969f172b7a3ea8928ed5fc348a8d24f755753efaf837/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~projected/kube-api-access-9fkth DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81aa7b94185e8fee5ca23b9951b10815e719e40dbb678f9864b4a7b7c1a3fb75/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a868da5249e9337fce3868b1cee5342e8174dcdcf92350cd7731fd6962778911/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8ae3e225-68cf-4814-8d33-e184ad008b3c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f486f2e8-4167-421a-8f04-5f6ce10b6c25/volumes/kubernetes.io~projected/kube-api-access-h44jz DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f93bd2e-a21e-4bc1-993d-117adc818868/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/412a8656-9cb2-4e67-9752-f179cd02219b/volumes/kubernetes.io~projected/kube-api-access-fsh6f DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61e96f32-b0b7-40b7-97e5-61ff980cc465/volumes/kubernetes.io~projected/kube-api-access-xq6qg DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d7f6695-43a2-4f59-83c5-ee24748be7f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:272 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b6b2f8ac93581fabff86263e76a497edf88643553f7bdbb21be6c141eba8861/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f7411dc-7fed-43b4-be33-7eca7c3144d5/volumes/kubernetes.io~projected/kube-api-access-hwwx2 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50bf088245ee40c3ff911a2ab28abebf7d947bf233935ac8ac5c1bfe1773bca8/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46e38d430c2355571ef9f9caf7880f5e9f0d7429f05bf2570ecfd30203bd2d97/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ff1f68b-203c-43a3-b5a5-671eda399b32/volumes/kubernetes.io~projected/kube-api-access-cwcc7 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c4a8839aa5b5c4ffcb98f8abf9842aac911cb4b2ad90e504992b154031e0ca1/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~projected/kube-api-access-4pm8d DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d2f4c5d-a352-4c80-8794-edf020ddbafa/volumes/kubernetes.io~projected/kube-api-access-44spc DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/65430e83-f633-41de-83c2-efc743b8f140/volumes/kubernetes.io~projected/kube-api-access-bkmts DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d9c9c95-575e-4d45-9416-2c8bd643a5e3/volumes/kubernetes.io~projected/kube-api-access-hvxwg DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29d6f07b-b204-4274-9d60-f6f7a32b4a90/volumes/kubernetes.io~projected/kube-api-access-gfr58 DeviceMajor:0 DeviceMinor:326 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c86bf44b6e52b88ccdc2da1cdf915d5bdc166534f3d32c87c3a28784b8ce64f/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95d06a2c-a395-419e-b1d4-2285608d71da/volumes/kubernetes.io~projected/kube-api-access-v9562 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2d50e209-0bda-4891-be9d-9e3580efab4e/volumes/kubernetes.io~projected/kube-api-access-898q4 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9e051aa-16cd-43b2-aec2-4f19f35ab485/volumes/kubernetes.io~projected/kube-api-access-pnhcb DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49/volumes/kubernetes.io~projected/kube-api-access-pcwwt DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b00097a4-0c42-452a-af4f-041cf40aecab/volumes/kubernetes.io~projected/kube-api-access-9vf5k DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbe5d4e726488f39e18e12a2dfb2c65308b4109d1b30bc793cbc7b26b8205382/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c86bf44b6e52b8 MacAddress:5a:ff:fd:39:ef:bb Speed:10000 Mtu:8901} {Name:3c5915954fd7b6e MacAddress:8a:5c:e9:85:8d:1f Speed:10000 Mtu:8901} {Name:46e38d430c23555 MacAddress:f6:d1:86:9c:d2:ae Speed:10000 Mtu:8901} {Name:585c65551856ee6 MacAddress:1a:45:7e:80:8a:ec Speed:10000 Mtu:8901} {Name:5bb599c421e7859 MacAddress:06:15:c2:3b:2f:17 Speed:10000 Mtu:8901} {Name:5c2b67459c06c5c MacAddress:92:6c:c9:fb:e2:d9 Speed:10000 Mtu:8901} {Name:7eb18f72c4a30e9 MacAddress:fa:83:d7:dc:be:57 Speed:10000 Mtu:8901} {Name:8b6b2f8ac93581f MacAddress:82:c9:54:22:f8:d3 Speed:10000 Mtu:8901} {Name:969f05b31414431 MacAddress:2a:21:83:8c:6e:1f Speed:10000 Mtu:8901} {Name:b5926140d68db04 MacAddress:7e:f0:47:d3:49:bc Speed:10000 Mtu:8901} {Name:bc15f483dae39cd MacAddress:1e:9a:f4:f7:89:58 Speed:10000 Mtu:8901} {Name:bf3aae969dbefcc MacAddress:8e:12:dc:7b:a9:89 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:c7:82:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:c2:ab:fa:c2:21:e0 Speed:0 Mtu:8901} {Name:e2dba5842d55c76 MacAddress:5a:91:d3:3b:44:c9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:c7:82:f9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:00:85:2e:a7:8c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:60:f6:90:1d:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.336546 4920 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.336590 4920 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.336922 4920 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.336944 4920 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337087 4920 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337095 4920 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337106 4920 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337120 4920 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:56.337132 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337137 4920 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337357 4920 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337473 4920 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337510 4920 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337526 4920 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337536 4920 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337545 4920 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:53:56.337550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.337553 4920 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:53:56.339755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.339062 4920 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:53:56.339755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.339084 4920 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:56.340345 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.340078 4920 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:53:56.342796 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.342778 4920 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:53:56.342796 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.342793 4920 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:56.342904 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.342895 4920 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 08:53:56.343117 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343106 4920 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:53:56.343153 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343135 4920 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:53:56.343246 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343237 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343252 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343259 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343265 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343271 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343277 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:53:56.343281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343282 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343288 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343295 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343301 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343310 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343320 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343348 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:53:56.343495 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.343353 4920 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:53:56.346820 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.346804 4920 server.go:1263] "Started kubelet" Oct 30 08:53:56.347299 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.346889 4920 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:53:56.347299 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.346955 4920 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:53:56.347299 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.347037 4920 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:53:56.347353 ip-10-0-129-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:53:56.347960 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.347733 4920 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:53:56.348500 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.348377 4920 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:53:56.351029 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.350998 4920 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:53:56.352621 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.351080 4920 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:56.352738 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.351193 4920 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Oct 30 08:53:56.352738 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.351247 4920 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:53:56.352738 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352684 4920 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:53:56.352738 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352701 4920 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:17 +0000 UTC" deadline="2028-07-31 08:07:06.253720172 +0000 UTC" Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352747 4920 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15359h13m9.900976336s" Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352595 4920 factory.go:55] Registering systemd factory Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352773 4920 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.351258 4920 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.352992 4920 factory.go:153] Registering CRI-O factory Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.353007 4920 factory.go:223] Registration of the crio container factory successfully Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.353064 4920 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.353085 4920 factory.go:103] Registering Raw factory Oct 30 08:53:56.353113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.353102 4920 manager.go:1199] Started watching for new ooms in manager Oct 30 08:53:56.353547 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.353533 4920 manager.go:319] Starting recovery of all containers Oct 30 08:53:56.360607 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360569 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d50e209-0bda-4891-be9d-9e3580efab4e" volumeName="kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360610 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ae3e225-68cf-4814-8d33-e184ad008b3c" volumeName="kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360633 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360653 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d06a2c-a395-419e-b1d4-2285608d71da" volumeName="kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360664 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e051aa-16cd-43b2-aec2-4f19f35ab485" volumeName="kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360674 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb312004-8950-4bea-ab8c-2c5279052edf" volumeName="kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360685 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="412a8656-9cb2-4e67-9752-f179cd02219b" volumeName="kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360694 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f486f2e8-4167-421a-8f04-5f6ce10b6c25" volumeName="kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360706 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb312004-8950-4bea-ab8c-2c5279052edf" volumeName="kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc" seLinuxMountContext="" Oct 30 08:53:56.360720 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360715 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b7f0d8-6100-45a8-84d1-41c8b5fbcb49" volumeName="kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360726 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360734 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adbbc457-38a1-45fd-ab3d-738f7dd07913" volumeName="kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360743 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360753 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360765 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a26a8219-f82f-408c-9241-a82eec0b312d" volumeName="kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360776 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a26a8219-f82f-408c-9241-a82eec0b312d" volumeName="kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360785 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61e96f32-b0b7-40b7-97e5-61ff980cc465" volumeName="kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360810 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e00d62-c126-48fb-aab2-d1eac679b710" volumeName="kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360823 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360835 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360846 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d06a2c-a395-419e-b1d4-2285608d71da" volumeName="kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360853 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f486f2e8-4167-421a-8f04-5f6ce10b6c25" volumeName="kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360864 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a50c02c-9976-4034-8785-68c3e971060a" volumeName="kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360874 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="627ed97e-e994-4439-815e-d7fe229402e5" volumeName="kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360881 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" volumeName="kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360891 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" volumeName="kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360900 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360911 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f381ca-b571-4170-a89c-ecdef16c72f6" volumeName="kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360922 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b" volumeName="kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360935 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ae3e225-68cf-4814-8d33-e184ad008b3c" volumeName="kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360943 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ae3e225-68cf-4814-8d33-e184ad008b3c" volumeName="kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360954 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adbbc457-38a1-45fd-ab3d-738f7dd07913" volumeName="kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360970 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b00097a4-0c42-452a-af4f-041cf40aecab" volumeName="kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360982 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e051aa-16cd-43b2-aec2-4f19f35ab485" volumeName="kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.360999 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361008 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d6f07b-b204-4274-9d60-f6f7a32b4a90" volumeName="kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361018 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d06a2c-a395-419e-b1d4-2285608d71da" volumeName="kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361026 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" volumeName="kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg" seLinuxMountContext="" Oct 30 08:53:56.361023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361035 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361050 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361059 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f381ca-b571-4170-a89c-ecdef16c72f6" volumeName="kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361071 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f381ca-b571-4170-a89c-ecdef16c72f6" volumeName="kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361078 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361089 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d06a2c-a395-419e-b1d4-2285608d71da" volumeName="kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361097 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adbbc457-38a1-45fd-ab3d-738f7dd07913" volumeName="kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361104 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361115 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d50e209-0bda-4891-be9d-9e3580efab4e" volumeName="kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361124 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b7f0d8-6100-45a8-84d1-41c8b5fbcb49" volumeName="kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361143 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" volumeName="kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361156 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95cb918-673c-4d8d-ac5e-3b8b2c38bab8" volumeName="kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361166 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9c9c95-575e-4d45-9416-2c8bd643a5e3" volumeName="kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361178 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361185 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49827cf2-721e-4b0e-b38b-ec64d7b0c9ed" volumeName="kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361203 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65430e83-f633-41de-83c2-efc743b8f140" volumeName="kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361225 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d7f6695-43a2-4f59-83c5-ee24748be7f5" volumeName="kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361238 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2f4c5d-a352-4c80-8794-edf020ddbafa" volumeName="kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361265 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169" volumeName="kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361273 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d06a2c-a395-419e-b1d4-2285608d71da" volumeName="kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361282 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361294 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e051aa-16cd-43b2-aec2-4f19f35ab485" volumeName="kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361306 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f486f2e8-4167-421a-8f04-5f6ce10b6c25" volumeName="kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361322 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ff1f68b-203c-43a3-b5a5-671eda399b32" volumeName="kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361335 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b" volumeName="kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361347 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361365 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361378 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e00d62-c126-48fb-aab2-d1eac679b710" volumeName="kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361396 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61e96f32-b0b7-40b7-97e5-61ff980cc465" volumeName="kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361408 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b7f0d8-6100-45a8-84d1-41c8b5fbcb49" volumeName="kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361425 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f7411dc-7fed-43b4-be33-7eca7c3144d5" volumeName="kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361438 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adbbc457-38a1-45fd-ab3d-738f7dd07913" volumeName="kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361450 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65430e83-f633-41de-83c2-efc743b8f140" volumeName="kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361468 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65430e83-f633-41de-83c2-efc743b8f140" volumeName="kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361485 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d2f4c5d-a352-4c80-8794-edf020ddbafa" volumeName="kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361502 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f93bd2e-a21e-4bc1-993d-117adc818868" volumeName="kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361515 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361528 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86dd71b-65b4-48bc-9b09-772f6dd7053a" volumeName="kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361547 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a50c02c-9976-4034-8785-68c3e971060a" volumeName="kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361562 4920 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25c4713f-3616-4d01-b7c7-310f70a14127" volumeName="kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca" seLinuxMountContext="" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361574 4920 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:53:56.362076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.361583 4920 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:53:56.378292 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.378264 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:56.378478 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.378461 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:56.388294 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.388263 4920 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:53:56.389696 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.389663 4920 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:53:56.389696 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.389688 4920 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:53:56.389849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.389709 4920 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:53:56.389849 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.389749 4920 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 08:53:56.391053 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.391021 4920 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 08:53:56.392238 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.392196 4920 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:56.392642 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.392621 4920 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:56.402815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.402790 4920 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:56.403051 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.403030 4920 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:56.406051 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.406029 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:56.406139 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.406065 4920 generic.go:356] "Generic (PLEG): container finished" podID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" exitCode=255 Oct 30 08:53:56.412437 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.412412 4920 generic.go:356] "Generic (PLEG): container finished" podID="e421a574dfe363a6bf33baf4cfc6af78" containerID="05796485adf348412b79a9cc895a2c916d40f0171af94c1ce26690764f60ed3a" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439638 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="2d65f32d730c349e3f3413de77efb876009236b270b2303c0c5f5a5cdea12ff3" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439662 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="080f002500c1ee10c9d30aa39897f1c61a8fc085db09732c2ea2f6c3ef44e620" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439668 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="64dcd806ed1f22b60ffeff15ac308b78c6df8e86bd5bcaa440c87e5154cbfb61" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439673 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="4622c349f627806aa4de03021c0b10fba80eb81ee04b67a9a7994371821bafa8" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439679 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="1d72d039d7bfdaa56c4c4af73fa0674e9d7c27265fa991758c90253471da9ac1" exitCode=0 Oct 30 08:53:56.439670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.439684 4920 generic.go:356] "Generic (PLEG): container finished" podID="d0a1416c-5f6d-4ac7-9f38-2760a05daa83" containerID="a885fc789c894502d1c51b207982f4a82bc605d68d7b8a11aa715abc07803765" exitCode=0 Oct 30 08:53:56.447739 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.447686 4920 manager.go:324] Recovery completed Oct 30 08:53:56.467439 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467418 4920 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:53:56.467439 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467434 4920 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:53:56.467591 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467456 4920 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:53:56.467639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467589 4920 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 08:53:56.467639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467600 4920 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 08:53:56.467639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467628 4920 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 08:53:56.467639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467633 4920 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 08:53:56.467639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467638 4920 policy_none.go:50] "Start" Oct 30 08:53:56.467763 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467654 4920 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:53:56.467763 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467663 4920 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:53:56.467819 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467795 4920 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 08:53:56.467819 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467803 4920 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 08:53:56.467819 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.467808 4920 policy_none.go:44] "Start" Oct 30 08:53:56.472601 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.472558 4920 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:53:56.477198 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.477179 4920 manager.go:341] "Starting Device Plugin manager" Oct 30 08:53:56.477761 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.477211 4920 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:53:56.477761 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.477762 4920 server.go:87] "Starting device plugin registration server" Oct 30 08:53:56.478021 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478007 4920 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:53:56.478074 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478022 4920 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:53:56.478206 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478185 4920 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:53:56.478343 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478311 4920 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:53:56.478415 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478352 4920 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:53:56.478415 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478360 4920 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:53:56.478510 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.478404 4920 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:53:56.478340748Z","UUID":"c1641081-6bb5-4b92-9a40-8f30e1d4d511","Handler":null,"Name":"","Endpoint":""} Oct 30 08:53:56.479725 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.479706 4920 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:53:56.480050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.480002 4920 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:53:56.480050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.480033 4920 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:53:56.489832 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.489798 4920 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal"] Oct 30 08:53:56.490417 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490381 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"ba69e635f74a36f196d077f731ece3db","Type":"ContainerStarted","Data":"341fd64337799d618e7d771e4cd20eedd72be6772d7fd50b38d9086f5a6224a6"} Oct 30 08:53:56.490417 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490418 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"ba69e635f74a36f196d077f731ece3db","Type":"ContainerStarted","Data":"4574ac4011d425bb0dbe278e5fc43bc69725a24491aa08579037d13b2ea1f370"} Oct 30 08:53:56.490599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490443 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerStarted","Data":"f8fa46300fa33afd16a35915fbeb2e7aefaca90ee8851c8485f755309b74428b"} Oct 30 08:53:56.490599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490455 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerDied","Data":"05796485adf348412b79a9cc895a2c916d40f0171af94c1ce26690764f60ed3a"} Oct 30 08:53:56.490599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490467 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"e421a574dfe363a6bf33baf4cfc6af78","Type":"ContainerStarted","Data":"b312ad96f622c32e531cc5e81c88945b1fab5236fb62bb5071605bc555453800"} Oct 30 08:53:56.490599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490513 4920 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.490599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.490513 4920 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.511986 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.511964 4920 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:56.512082 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.512007 4920 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.512342 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.512328 4920 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:56.512381 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:56.512365 4920 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.578870 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.578812 4920 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:56.580697 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.580678 4920 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:56.580815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.580707 4920 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:56.580815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.580719 4920 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:56.580815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.580770 4920 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.591684 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.591651 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.591684 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.591684 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.591833 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.591702 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.612499 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.612469 4920 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.612579 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.612544 4920 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692325 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692361 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692569 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692379 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692569 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692430 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692569 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692437 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba69e635f74a36f196d077f731ece3db-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"ba69e635f74a36f196d077f731ece3db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:56.692569 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:56.692482 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e421a574dfe363a6bf33baf4cfc6af78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"e421a574dfe363a6bf33baf4cfc6af78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:57.337902 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.337868 4920 apiserver.go:52] "Watching apiserver" Oct 30 08:53:57.383054 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.383021 4920 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:53:57.383733 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.383708 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ccr66","openshift-network-console/networking-console-plugin-86c478bb88-4p7wn","openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg","openshift-multus/multus-additional-cni-plugins-5rc7p","kube-system/konnectivity-agent-64jq8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4","openshift-network-diagnostics/network-check-source-6f78d59f99-477hl","openshift-ovn-kubernetes/ovnkube-node-86p9d","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal","openshift-image-registry/node-ca-2zzzz","openshift-network-operator/iptables-alerter-pkr5v","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr","openshift-console-operator/console-operator-7cd56b8dbc-9mxzf","openshift-insights/insights-operator-7c4986d449-tn25f","openshift-multus/multus-lvb22","openshift-cluster-node-tuning-operator/tuned-lzhf7","openshift-insights/insights-runtime-extractor-sxh78","openshift-network-diagnostics/network-check-target-xb9sq","openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2","openshift-ingress-canary/ingress-canary-vbc5k","openshift-ingress/router-default-7888d9c8bc-nzrhq","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq","openshift-dns/node-resolver-l6qzz","openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x","kube-system/global-pull-secret-syncer-p9764","openshift-multus/network-metrics-daemon-zqlwh","openshift-image-registry/image-registry-65ffb76754-ndfbg","kube-system/kas-connection-checker-685895c7c9-8lfmm","openshift-dns/dns-default-mj9w5","kube-system/kas-connection-checker-685895c7c9-hhlxw","openshift-image-registry/image-registry-fd88cff75-jmrdx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal"] Oct 30 08:53:57.385115 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.385093 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.386295 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.386278 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.388261 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.388238 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.389376 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.389355 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.391150 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.391131 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:57.396498 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.396479 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:57.397479 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.397464 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.398006 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.397992 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.398953 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.398925 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.398953 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.398938 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 08:53:57.399136 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399117 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399226 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399200 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 08:53:57.399285 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399206 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399285 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399253 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399285 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399255 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.399285 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399280 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 08:53:57.399496 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399349 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 08:53:57.399496 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399395 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399496 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399399 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 08:53:57.399639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399514 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399525 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.399639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399564 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 08:53:57.399639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399596 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399665 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399678 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399699 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399735 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399751 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399759 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399769 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399796 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 08:53:57.399859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.399800 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m8kth\"" Oct 30 08:53:57.400433 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.400409 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.400522 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.400503 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.400613 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.400578 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 08:53:57.400708 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.400612 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.400708 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.400641 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4wbtr\"" Oct 30 08:53:57.401113 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.401096 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 08:53:57.409156 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.409129 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wkc6q\"" Oct 30 08:53:57.410205 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.410189 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 08:53:57.410829 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.410810 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 08:53:57.410909 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.410864 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 08:53:57.410909 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.410883 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.411134 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.411122 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.412082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.412049 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 08:53:57.412447 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.412428 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 08:53:57.412544 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.412520 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 08:53:57.413627 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.413608 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:53:57.413752 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.413737 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.413815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.413765 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:53:57.413815 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.413773 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:53:57.414016 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.414002 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.414096 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.414024 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:53:57.414096 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.414064 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:53:57.414183 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.414113 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.419100 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.419084 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w9gxh\"" Oct 30 08:53:57.420745 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420729 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 08:53:57.420807 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420772 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.420866 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420817 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:53:57.420866 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420858 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:53:57.420962 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420903 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.420962 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420917 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:53:57.420962 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420931 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:53:57.421272 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420932 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421268 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420973 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421321 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420982 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421022 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421034 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421039 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421048 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421361 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421173 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421174 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421202 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421242 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dfndm\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.420972 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421076 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421113 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421459 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421464 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421477 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421515 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421558 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421564 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421573 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421589 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421601 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421606 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421615 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421628 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421626 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421658 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421670 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421695 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421728 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421754 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421771 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c7tn4\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421781 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.421801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421790 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421827 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421857 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h5xkm\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421884 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421905 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421932 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421941 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.421944 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422037 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422114 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422164 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422204 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-rm2lv\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422307 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:53:57.422605 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.422451 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 08:53:57.443128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.443103 4920 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:57.443276 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.443263 4920 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:57.455574 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.455552 4920 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:53:57.463736 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.463718 4920 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:57.463841 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.463771 4920 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Oct 30 08:53:57.464408 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.464395 4920 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:57.464483 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.464431 4920 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Oct 30 08:53:57.496006 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.495975 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.496171 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496013 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.496171 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496038 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.496171 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496062 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496183 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496211 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496258 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496285 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496307 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496327 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-service-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496335 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.496369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496363 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496392 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496413 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496420 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-script-lib\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496442 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47f381ca-b571-4170-a89c-ecdef16c72f6-config\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496423 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496495 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjdg\" (UniqueName: \"kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496506 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496554 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496558 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496575 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d7f6695-43a2-4f59-83c5-ee24748be7f5-original-pull-secret\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496606 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496644 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496656 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-tmp\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496671 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496700 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxwg\" (UniqueName: \"kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg\") pod \"kas-connection-checker-685895c7c9-ccr66\" (UID: \"0d9c9c95-575e-4d45-9416-2c8bd643a5e3\") " pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496659 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-daemon-config\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496740 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.496755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496745 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44jz\" (UniqueName: \"kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496773 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496800 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496826 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk7n\" (UniqueName: \"kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n\") pod \"kas-connection-checker-685895c7c9-hhlxw\" (UID: \"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169\") " pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496866 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496906 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496931 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496942 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f93bd2e-a21e-4bc1-993d-117adc818868-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.496982 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr58\" (UniqueName: \"kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58\") pod \"migrator-66df5d496b-4595x\" (UID: \"29d6f07b-b204-4274-9d60-f6f7a32b4a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497019 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497063 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497094 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497098 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497150 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-env-overrides\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497166 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497263 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497318 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497345 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497348 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497390 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7r7\" (UniqueName: \"kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.497427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497417 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497441 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497467 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497492 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497536 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497560 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-default-certificate\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497582 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497618 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-tmp\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497631 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497658 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497692 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497712 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47f381ca-b571-4170-a89c-ecdef16c72f6-serving-cert\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497732 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497754 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497771 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497802 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497822 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497837 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497852 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497865 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cni-binary-copy\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497874 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497902 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6q6\" (UniqueName: \"kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497917 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f93bd2e-a21e-4bc1-993d-117adc818868-snapshots\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497918 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsh6f\" (UniqueName: \"kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f\") pod \"volume-data-source-validator-58c9f5f46d-8jhj4\" (UID: \"412a8656-9cb2-4e67-9752-f179cd02219b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.497977 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498060 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95d06a2c-a395-419e-b1d4-2285608d71da-ovn-node-metrics-cert\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498064 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498118 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498123 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.498149 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498157 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498205 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498268 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498298 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhcb\" (UniqueName: \"kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498325 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498325 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e051aa-16cd-43b2-aec2-4f19f35ab485-config-volume\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498350 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498366 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ae3e225-68cf-4814-8d33-e184ad008b3c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498470 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d2f4c5d-a352-4c80-8794-edf020ddbafa-serviceca\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498506 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498518 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a50c02c-9976-4034-8785-68c3e971060a-agent-certs\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498527 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498561 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498590 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498606 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9562\" (UniqueName: \"kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498621 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498646 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498663 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498693 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498733 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498813 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adbbc457-38a1-45fd-ab3d-738f7dd07913-serving-cert\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498830 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498847 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e00d62-c126-48fb-aab2-d1eac679b710-cert\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498895 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498900 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2d50e209-0bda-4891-be9d-9e3580efab4e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498896 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498924 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.498955 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499002 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmts\" (UniqueName: \"kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499035 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h26\" (UniqueName: \"kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26\") pod \"kas-connection-checker-685895c7c9-8lfmm\" (UID: \"627ed97e-e994-4439-815e-d7fe229402e5\") " pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499057 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499083 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499101 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499124 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.499293 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499153 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcc7\" (UniqueName: \"kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499328 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499351 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499407 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499425 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499441 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499462 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499478 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnnc\" (UniqueName: \"kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499509 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499556 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499573 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499605 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499622 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8xk\" (UniqueName: \"kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499660 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499684 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/adbbc457-38a1-45fd-ab3d-738f7dd07913-config\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499697 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwwt\" (UniqueName: \"kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499736 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65430e83-f633-41de-83c2-efc743b8f140-etc-tuned\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499740 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499799 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499809 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499842 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499846 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499884 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499901 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499918 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbwb\" (UniqueName: \"kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499944 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499973 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e96f32-b0b7-40b7-97e5-61ff980cc465-tmp-dir\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.499979 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw65l\" (UniqueName: \"kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500034 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500062 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500089 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500117 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500140 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500157 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500188 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwx2\" (UniqueName: \"kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500201 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb312004-8950-4bea-ab8c-2c5279052edf-iptables-alerter-script\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500275 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500309 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500335 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500372 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500413 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500418 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500457 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500483 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500502 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500520 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500528 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e051aa-16cd-43b2-aec2-4f19f35ab485-tmp-dir\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500535 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500593 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-data-volume\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500591 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.500616 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500633 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500668 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500696 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500701 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-stats-auth\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500738 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500821 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ae3e225-68cf-4814-8d33-e184ad008b3c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.500772 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501172 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501190 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501208 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898q4\" (UniqueName: \"kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501247 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501271 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501366 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501400 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f93bd2e-a21e-4bc1-993d-117adc818868-serving-cert\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501429 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501468 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6qg\" (UniqueName: \"kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501476 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a50c02c-9976-4034-8785-68c3e971060a-konnectivity-ca\") pod \"konnectivity-agent-64jq8\" (UID: \"0a50c02c-9976-4034-8785-68c3e971060a\") " pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501501 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501522 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt7w\" (UniqueName: \"kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w\") pod \"network-check-source-6f78d59f99-477hl\" (UID: \"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501555 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501564 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501605 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44spc\" (UniqueName: \"kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501632 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501664 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501689 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501719 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501744 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.501988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.501867 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95d06a2c-a395-419e-b1d4-2285608d71da-ovnkube-config\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512516 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxwg\" (UniqueName: \"kubernetes.io/projected/0d9c9c95-575e-4d45-9416-2c8bd643a5e3-kube-api-access-hvxwg\") pod \"kas-connection-checker-685895c7c9-ccr66\" (UID: \"0d9c9c95-575e-4d45-9416-2c8bd643a5e3\") " pod="kube-system/kas-connection-checker-685895c7c9-ccr66" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512731 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr58\" (UniqueName: \"kubernetes.io/projected/29d6f07b-b204-4274-9d60-f6f7a32b4a90-kube-api-access-gfr58\") pod \"migrator-66df5d496b-4595x\" (UID: \"29d6f07b-b204-4274-9d60-f6f7a32b4a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4595x" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512755 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44jz\" (UniqueName: \"kubernetes.io/projected/f486f2e8-4167-421a-8f04-5f6ce10b6c25-kube-api-access-h44jz\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512903 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk7n\" (UniqueName: \"kubernetes.io/projected/fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169-kube-api-access-mwk7n\") pod \"kas-connection-checker-685895c7c9-hhlxw\" (UID: \"fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169\") " pod="kube-system/kas-connection-checker-685895c7c9-hhlxw" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512913 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjdg\" (UniqueName: \"kubernetes.io/projected/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-kube-api-access-whjdg\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512941 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6q6\" (UniqueName: \"kubernetes.io/projected/8ae3e225-68cf-4814-8d33-e184ad008b3c-kube-api-access-xc6q6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr\" (UID: \"8ae3e225-68cf-4814-8d33-e184ad008b3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" Oct 30 08:53:57.512956 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.512952 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7r7\" (UniqueName: \"kubernetes.io/projected/adbbc457-38a1-45fd-ab3d-738f7dd07913-kube-api-access-5r7r7\") pod \"console-operator-7cd56b8dbc-9mxzf\" (UID: \"adbbc457-38a1-45fd-ab3d-738f7dd07913\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:57.513253 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.513134 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsh6f\" (UniqueName: \"kubernetes.io/projected/412a8656-9cb2-4e67-9752-f179cd02219b-kube-api-access-fsh6f\") pod \"volume-data-source-validator-58c9f5f46d-8jhj4\" (UID: \"412a8656-9cb2-4e67-9752-f179cd02219b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-8jhj4" Oct 30 08:53:57.517455 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.517433 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.517643 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.517622 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcc7\" (UniqueName: \"kubernetes.io/projected/0ff1f68b-203c-43a3-b5a5-671eda399b32-kube-api-access-cwcc7\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:57.517707 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.517660 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/a26a8219-f82f-408c-9241-a82eec0b312d-kube-api-access-2sswm\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.517879 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.517862 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9562\" (UniqueName: \"kubernetes.io/projected/95d06a2c-a395-419e-b1d4-2285608d71da-kube-api-access-v9562\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.518152 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.518128 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhcb\" (UniqueName: \"kubernetes.io/projected/c9e051aa-16cd-43b2-aec2-4f19f35ab485-kube-api-access-pnhcb\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.518244 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.518149 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnnc\" (UniqueName: \"kubernetes.io/projected/fb312004-8950-4bea-ab8c-2c5279052edf-kube-api-access-bmnnc\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.518298 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.518281 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkth\" (UniqueName: \"kubernetes.io/projected/8f93bd2e-a21e-4bc1-993d-117adc818868-kube-api-access-9fkth\") pod \"insights-operator-7c4986d449-tn25f\" (UID: \"8f93bd2e-a21e-4bc1-993d-117adc818868\") " pod="openshift-insights/insights-operator-7c4986d449-tn25f" Oct 30 08:53:57.519792 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.519772 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6qg\" (UniqueName: \"kubernetes.io/projected/61e96f32-b0b7-40b7-97e5-61ff980cc465-kube-api-access-xq6qg\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.519867 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.519827 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.519867 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.519832 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwwt\" (UniqueName: \"kubernetes.io/projected/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-kube-api-access-pcwwt\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.520196 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.520181 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt7w\" (UniqueName: \"kubernetes.io/projected/49827cf2-721e-4b0e-b38b-ec64d7b0c9ed-kube-api-access-5zt7w\") pod \"network-check-source-6f78d59f99-477hl\" (UID: \"49827cf2-721e-4b0e-b38b-ec64d7b0c9ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-477hl" Oct 30 08:53:57.522653 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.522620 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44spc\" (UniqueName: \"kubernetes.io/projected/8d2f4c5d-a352-4c80-8794-edf020ddbafa-kube-api-access-44spc\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.522865 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.522845 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8xk\" (UniqueName: \"kubernetes.io/projected/11e00d62-c126-48fb-aab2-d1eac679b710-kube-api-access-pf8xk\") pod \"ingress-canary-vbc5k\" (UID: \"11e00d62-c126-48fb-aab2-d1eac679b710\") " pod="openshift-ingress-canary/ingress-canary-vbc5k" Oct 30 08:53:57.522999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.522977 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwx2\" (UniqueName: \"kubernetes.io/projected/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kube-api-access-hwwx2\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.523191 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.523157 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmts\" (UniqueName: \"kubernetes.io/projected/65430e83-f633-41de-83c2-efc743b8f140-kube-api-access-bkmts\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.523322 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.523305 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf5k\" (UniqueName: \"kubernetes.io/projected/b00097a4-0c42-452a-af4f-041cf40aecab-kube-api-access-9vf5k\") pod \"network-check-target-xb9sq\" (UID: \"b00097a4-0c42-452a-af4f-041cf40aecab\") " pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:53:57.523434 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.523414 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.523855 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.523840 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-898q4\" (UniqueName: \"kubernetes.io/projected/2d50e209-0bda-4891-be9d-9e3580efab4e-kube-api-access-898q4\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.525328 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.525306 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h26\" (UniqueName: \"kubernetes.io/projected/627ed97e-e994-4439-815e-d7fe229402e5-kube-api-access-z8h26\") pod \"kas-connection-checker-685895c7c9-8lfmm\" (UID: \"627ed97e-e994-4439-815e-d7fe229402e5\") " pod="kube-system/kas-connection-checker-685895c7c9-8lfmm" Oct 30 08:53:57.525505 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.525487 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbwb\" (UniqueName: \"kubernetes.io/projected/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-api-access-brbwb\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.525540 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.525487 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.525670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.525655 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw65l\" (UniqueName: \"kubernetes.io/projected/47f381ca-b571-4170-a89c-ecdef16c72f6-kube-api-access-qw65l\") pod \"service-ca-operator-544d749c8c-nnx9z\" (UID: \"47f381ca-b571-4170-a89c-ecdef16c72f6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" Oct 30 08:53:57.602832 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602798 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.602832 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602831 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602847 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602862 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602918 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602930 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602950 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602956 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-multus\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602972 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602994 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602995 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-kubelet-config\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.603010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603011 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602960 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-netd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603033 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.602923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-host\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603068 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-kubelet\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603088 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603092 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d7f6695-43a2-4f59-83c5-ee24748be7f5-dbus\") pod \"global-pull-secret-syncer-p9764\" (UID: \"7d7f6695-43a2-4f59-83c5-ee24748be7f5\") " pod="kube-system/global-pull-secret-syncer-p9764" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603106 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603137 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603148 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-cnibin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603158 4920 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603162 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603193 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-systemd\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603196 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603242 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.103201752 +0000 UTC m=+1.826628888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603276 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-systemd-units\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603285 4920 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603295 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.103273514 +0000 UTC m=+1.826700637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : secret "metrics-daemon-secret" not found Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603376 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603405 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603418 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.103405814 +0000 UTC m=+1.826832940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603419 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-system-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603443 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-etc-kubernetes\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.603470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603469 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603488 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603522 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603539 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603547 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603555 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603521 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-slash\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603555 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-cni-bin\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603584 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603593 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-etc-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603605 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-ovn\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603608 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603635 4920 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603645 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603667 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603686 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-multus-certs\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603688 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603711 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert podName:e95cb918-673c-4d8d-ac5e-3b8b2c38bab8 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.103696929 +0000 UTC m=+1.827124058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4p7wn" (UID: "e95cb918-673c-4d8d-ac5e-3b8b2c38bab8") : secret "networking-console-plugin-cert" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603712 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-lib-modules\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603717 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-netns\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603721 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-cni-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603734 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603761 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603774 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-var-lib-cni-bin\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603779 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603782 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-conf\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603795 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603813 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603817 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb312004-8950-4bea-ab8c-2c5279052edf-host-slash\") pod \"iptables-alerter-pkr5v\" (UID: \"fb312004-8950-4bea-ab8c-2c5279052edf\") " pod="openshift-network-operator/iptables-alerter-pkr5v" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603830 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-var-lib-openvswitch\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603832 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603853 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-socket-dir-parent\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603843 4920 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603867 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.603888 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.10387982 +0000 UTC m=+1.827306974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603901 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-modprobe-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603909 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-node-log\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603901 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603948 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603924 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysconfig\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603972 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603996 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-device-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.603997 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604025 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604052 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604057 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-multus-conf-dir\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604086 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604104 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604139 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-registration-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604169 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604195 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604235 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-system-cni-dir\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604238 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604268 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604272 4920 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.604281 4920 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604292 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604315 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:57.604337 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.104323707 +0000 UTC m=+1.827750848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604340 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-socket-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604358 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-var-lib-kubelet\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604366 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604423 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604336 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d2f4c5d-a352-4c80-8794-edf020ddbafa-host\") pod \"node-ca-2zzzz\" (UID: \"8d2f4c5d-a352-4c80-8794-edf020ddbafa\") " pod="openshift-image-registry/node-ca-2zzzz" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604446 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604475 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-run-netns\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.604430 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604482 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604492 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-run-systemd\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604507 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604513 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a1416c-5f6d-4ac7-9f38-2760a05daa83-cnibin\") pod \"multus-additional-cni-plugins-5rc7p\" (UID: \"d0a1416c-5f6d-4ac7-9f38-2760a05daa83\") " pod="openshift-multus/multus-additional-cni-plugins-5rc7p" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604549 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604563 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-kubernetes\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604551 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604576 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-run\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604619 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604643 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604658 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e96f32-b0b7-40b7-97e5-61ff980cc465-hosts-file\") pod \"node-resolver-l6qzz\" (UID: \"61e96f32-b0b7-40b7-97e5-61ff980cc465\") " pod="openshift-dns/node-resolver-l6qzz" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604670 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604682 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-sys\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604696 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604713 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-host-kubelet\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604718 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604722 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-host-run-k8s-cni-cncf-io\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604738 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85b7f0d8-6100-45a8-84d1-41c8b5fbcb49-hostroot\") pod \"multus-lvb22\" (UID: \"85b7f0d8-6100-45a8-84d1-41c8b5fbcb49\") " pod="openshift-multus/multus-lvb22" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604747 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604767 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95d06a2c-a395-419e-b1d4-2285608d71da-log-socket\") pod \"ovnkube-node-86p9d\" (UID: \"95d06a2c-a395-419e-b1d4-2285608d71da\") " pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604781 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604808 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65430e83-f633-41de-83c2-efc743b8f140-etc-sysctl-d\") pod \"tuned-lzhf7\" (UID: \"65430e83-f633-41de-83c2-efc743b8f140\") " pod="openshift-cluster-node-tuning-operator/tuned-lzhf7" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604831 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604845 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-sys-fs\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604865 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604872 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f7411dc-7fed-43b4-be33-7eca7c3144d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzxgg\" (UID: \"8f7411dc-7fed-43b4-be33-7eca7c3144d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzxgg" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.604956 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-crio-socket\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:57.606617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.606488 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f486f2e8-4167-421a-8f04-5f6ce10b6c25-metrics-certs\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:57.607675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.606955 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9e051aa-16cd-43b2-aec2-4f19f35ab485-metrics-tls\") pod \"dns-default-mj9w5\" (UID: \"c9e051aa-16cd-43b2-aec2-4f19f35ab485\") " pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:57.607675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.607035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"image-registry-fd88cff75-jmrdx\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.607675 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.607291 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"image-registry-65ffb76754-ndfbg\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.685108 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.685079 4920 scope.go:122] "RemoveContainer" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" Oct 30 08:53:57.688805 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.688784 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:57.698395 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.698375 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:53:57.698482 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:57.698416 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj9w5" Oct 30 08:53:58.025531 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.025467 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:53:58.028128 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:58.028099 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c4713f_3616_4d01_b7c7_310f70a14127.slice/crio-abb957b00b0f36212e8eddea2cc26c022dcabdec4a2b86795552ca59b5817980 WatchSource:0}: Error finding container abb957b00b0f36212e8eddea2cc26c022dcabdec4a2b86795552ca59b5817980: Status 404 returned error can't find the container with id abb957b00b0f36212e8eddea2cc26c022dcabdec4a2b86795552ca59b5817980 Oct 30 08:53:58.031965 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.031950 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zzzz_8d2f4c5d-a352-4c80-8794-edf020ddbafa/node-ca/0.log" Oct 30 08:53:58.059818 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.059785 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:53:58.070358 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:58.070322 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86dd71b_65b4_48bc_9b09_772f6dd7053a.slice/crio-4e0162dc7fa81fc0843630009593cb8ad773bb0ea1f4e220b1c6627d1b1f6819 WatchSource:0}: Error finding container 4e0162dc7fa81fc0843630009593cb8ad773bb0ea1f4e220b1c6627d1b1f6819: Status 404 returned error can't find the container with id 4e0162dc7fa81fc0843630009593cb8ad773bb0ea1f4e220b1c6627d1b1f6819 Oct 30 08:53:58.087888 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.087865 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mj9w5"] Oct 30 08:53:58.098249 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:58.098202 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e051aa_16cd_43b2_aec2_4f19f35ab485.slice/crio-6afaa471ef9548af03ff9c21605a3ff2a44c3a9cc09dd8df0934b9c72bc4fdfd WatchSource:0}: Error finding container 6afaa471ef9548af03ff9c21605a3ff2a44c3a9cc09dd8df0934b9c72bc4fdfd: Status 404 returned error can't find the container with id 6afaa471ef9548af03ff9c21605a3ff2a44c3a9cc09dd8df0934b9c72bc4fdfd Oct 30 08:53:58.100487 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.100470 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:53:58.109547 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109522 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:58.109643 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109564 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:58.109643 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109609 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:58.109643 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109624 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:58.109643 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109643 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109678 4920 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.109675 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109727 4920 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109747 4920 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109785 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:53:59.109742465 +0000 UTC m=+2.833169601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109804 4920 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:53:58.109838 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109807 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:53:59.109796352 +0000 UTC m=+2.833223475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:58.110051 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109854 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:59.10983879 +0000 UTC m=+2.833265924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:58.110051 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109866 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:53:59.109859485 +0000 UTC m=+2.833286607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:58.110051 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.109879 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:59.109870616 +0000 UTC m=+2.833297737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : secret "metrics-daemon-secret" not found Oct 30 08:53:58.111952 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.111934 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e95cb918-673c-4d8d-ac5e-3b8b2c38bab8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4p7wn\" (UID: \"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:58.278600 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.278518 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:58.291634 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.291609 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" Oct 30 08:53:58.378672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.378637 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:58.448450 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.448420 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/2.log" Oct 30 08:53:58.448858 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.448843 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/1.log" Oct 30 08:53:58.448947 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.448876 4920 generic.go:356] "Generic (PLEG): container finished" podID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" exitCode=255 Oct 30 08:53:58.449018 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.448953 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerDied","Data":"83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957"} Oct 30 08:53:58.449018 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.449008 4920 scope.go:122] "RemoveContainer" containerID="870753ca8e887fae01dafeea1d164ffe6c679a58311e649a10f538850aece4d8" Oct 30 08:53:58.449377 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.449358 4920 scope.go:122] "RemoveContainer" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" Oct 30 08:53:58.449572 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:58.449550 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:53:58.450468 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.450441 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj9w5" event={"ID":"c9e051aa-16cd-43b2-aec2-4f19f35ab485","Type":"ContainerStarted","Data":"6afaa471ef9548af03ff9c21605a3ff2a44c3a9cc09dd8df0934b9c72bc4fdfd"} Oct 30 08:53:58.452123 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.452097 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" event={"ID":"c86dd71b-65b4-48bc-9b09-772f6dd7053a","Type":"ContainerStarted","Data":"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee"} Oct 30 08:53:58.452254 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.452141 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" event={"ID":"c86dd71b-65b4-48bc-9b09-772f6dd7053a","Type":"ContainerStarted","Data":"4e0162dc7fa81fc0843630009593cb8ad773bb0ea1f4e220b1c6627d1b1f6819"} Oct 30 08:53:58.453891 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.453866 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" event={"ID":"25c4713f-3616-4d01-b7c7-310f70a14127","Type":"ContainerStarted","Data":"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6"} Oct 30 08:53:58.453992 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.453909 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" event={"ID":"25c4713f-3616-4d01-b7c7-310f70a14127","Type":"ContainerStarted","Data":"abb957b00b0f36212e8eddea2cc26c022dcabdec4a2b86795552ca59b5817980"} Oct 30 08:53:58.596028 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.595998 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4p7wn"] Oct 30 08:53:58.600156 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:53:58.600127 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95cb918_673c_4d8d_ac5e_3b8b2c38bab8.slice/crio-b2e60c4e456de3b1dd987805437e473c67978bca069e1bf975f79fd51884cc69 WatchSource:0}: Error finding container b2e60c4e456de3b1dd987805437e473c67978bca069e1bf975f79fd51884cc69: Status 404 returned error can't find the container with id b2e60c4e456de3b1dd987805437e473c67978bca069e1bf975f79fd51884cc69 Oct 30 08:53:58.719845 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.719805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:58.734647 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.734603 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:53:58.884869 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:58.884790 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:59.030850 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.030805 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbc5k_11e00d62-c126-48fb-aab2-d1eac679b710/serve-healthcheck-canary/0.log" Oct 30 08:53:59.075128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.075085 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:59.120163 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.120124 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:53:59.120339 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.120235 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:53:59.120339 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.120264 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:53:59.120339 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.120293 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:53:59.120339 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120309 4920 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:59.120339 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.120336 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120385 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.120363697 +0000 UTC m=+4.843790831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120386 4920 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120443 4920 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120458 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.120439927 +0000 UTC m=+4.843867059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : secret "metrics-daemon-secret" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120500 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls podName:a26a8219-f82f-408c-9241-a82eec0b312d nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.120485442 +0000 UTC m=+4.843912563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f2bwq" (UID: "a26a8219-f82f-408c-9241-a82eec0b312d") : secret "samples-operator-tls" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120527 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.120519967 +0000 UTC m=+4.843947088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120558 4920 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:59.120606 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.120601 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.120589725 +0000 UTC m=+4.844016864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:53:59.230679 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.230582 4920 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:53:59.232087 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.232063 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4595x_29d6f07b-b204-4274-9d60-f6f7a32b4a90/migrator/0.log" Oct 30 08:53:59.271689 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.271655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:59.393652 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.393596 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:59.460750 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.460723 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/2.log" Oct 30 08:53:59.461134 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.461114 4920 scope.go:122] "RemoveContainer" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" Oct 30 08:53:59.461385 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:53:59.461358 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:53:59.462846 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.462817 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" event={"ID":"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8","Type":"ContainerStarted","Data":"b2e60c4e456de3b1dd987805437e473c67978bca069e1bf975f79fd51884cc69"} Oct 30 08:53:59.462932 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.462891 4920 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:59.632753 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:53:59.632726 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr_8ae3e225-68cf-4814-8d33-e184ad008b3c/kube-storage-version-migrator-operator/0.log" Oct 30 08:54:00.468395 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.468359 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj9w5" event={"ID":"c9e051aa-16cd-43b2-aec2-4f19f35ab485","Type":"ContainerStarted","Data":"71ad9779dd9330c3b2c311b8aeec295ad5a0c8a1a838652d99408536d360ad11"} Oct 30 08:54:00.468902 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.468415 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj9w5" event={"ID":"c9e051aa-16cd-43b2-aec2-4f19f35ab485","Type":"ContainerStarted","Data":"0c2ac54c61a8ea5c773fc05c91b456207797e431e4b0551013de502fdf00a7c5"} Oct 30 08:54:00.468902 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.468441 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mj9w5" Oct 30 08:54:00.469856 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.469828 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4p7wn" event={"ID":"e95cb918-673c-4d8d-ac5e-3b8b2c38bab8","Type":"ContainerStarted","Data":"ff28936c1bd9ac44d099367fe1cb510dd1a24f7c09bffdcce1b20c056e0369a7"} Oct 30 08:54:00.478993 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.478968 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:00.676527 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.676492 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:00.938359 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:00.938329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:01.087452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.087417 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:01.138466 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.138419 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:54:01.138592 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.138510 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:54:01.138592 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.138532 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:01.138592 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.138553 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:54:01.138592 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138566 4920 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138639 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:54:05.138618315 +0000 UTC m=+8.862045435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138643 4920 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138653 4920 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138686 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle podName:f486f2e8-4167-421a-8f04-5f6ce10b6c25 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:05.138669093 +0000 UTC m=+8.862096235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle") pod "router-default-7888d9c8bc-nzrhq" (UID: "f486f2e8-4167-421a-8f04-5f6ce10b6c25") : configmap references non-existent config key: service-ca.crt Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138710 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:54:05.138701582 +0000 UTC m=+8.862128707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.138726 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:54:01.138793 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.138774 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs podName:0ff1f68b-203c-43a3-b5a5-671eda399b32 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:05.138767182 +0000 UTC m=+8.862194305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs") pod "network-metrics-daemon-zqlwh" (UID: "0ff1f68b-203c-43a3-b5a5-671eda399b32") : secret "metrics-daemon-secret" not found Oct 30 08:54:01.141052 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.141025 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a26a8219-f82f-408c-9241-a82eec0b312d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f2bwq\" (UID: \"a26a8219-f82f-408c-9241-a82eec0b312d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:54:01.287064 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.287036 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:01.289947 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.289929 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" Oct 30 08:54:01.584055 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.583975 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq"] Oct 30 08:54:01.592171 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.592148 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:54:01.592562 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.592544 4920 scope.go:122] "RemoveContainer" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" Oct 30 08:54:01.592875 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:01.592847 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:54:01.861127 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:01.861055 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:54:02.454652 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:02.454610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:02.479550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:02.479511 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" event={"ID":"a26a8219-f82f-408c-9241-a82eec0b312d","Type":"ContainerStarted","Data":"fc9ed632c5ffae9e11ce0d525e403084fb5e4310ab5eca013db4c909d300b9a3"} Oct 30 08:54:02.531570 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:02.531532 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:02.673977 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:02.673941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:02.876375 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:02.876344 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:03.074006 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:03.073948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:03.273768 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:03.273742 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:03.485148 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:03.485055 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" event={"ID":"a26a8219-f82f-408c-9241-a82eec0b312d","Type":"ContainerStarted","Data":"5fcf2950a92c4091ad96494865f6ccf8a6f1d02870ac08e57d70287fc692b7ac"} Oct 30 08:54:03.485148 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:03.485111 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f2bwq" event={"ID":"a26a8219-f82f-408c-9241-a82eec0b312d","Type":"ContainerStarted","Data":"31088cd8f760c0ec4dc6809a30a293dacfc6864f7038c88cb697f8679447775e"} Oct 30 08:54:03.730262 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:03.730209 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:54:04.471094 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:04.471057 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:04.673410 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:04.673373 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:04.877837 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:04.877800 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:05.073579 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.073546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:05.166004 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.165929 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:05.174025 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.174001 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:54:05.174104 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.174056 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:54:05.174104 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.174076 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:05.174212 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:05.174176 4920 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:05.174287 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.174234 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:54:05.174287 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:05.174268 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls podName:4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b nodeName:}" failed. No retries permitted until 2026-10-30 08:54:13.174246187 +0000 UTC m=+16.897673325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxh78" (UID: "4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:05.174385 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:05.174305 4920 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 08:54:05.174385 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:05.174350 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls podName:2d50e209-0bda-4891-be9d-9e3580efab4e nodeName:}" failed. No retries permitted until 2026-10-30 08:54:13.174335509 +0000 UTC m=+16.897762633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-stsh2" (UID: "2d50e209-0bda-4891-be9d-9e3580efab4e") : secret "cluster-monitoring-operator-tls" not found Oct 30 08:54:05.174783 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.174759 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f486f2e8-4167-421a-8f04-5f6ce10b6c25-service-ca-bundle\") pod \"router-default-7888d9c8bc-nzrhq\" (UID: \"f486f2e8-4167-421a-8f04-5f6ce10b6c25\") " pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:05.176415 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.176389 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff1f68b-203c-43a3-b5a5-671eda399b32-metrics-certs\") pod \"network-metrics-daemon-zqlwh\" (UID: \"0ff1f68b-203c-43a3-b5a5-671eda399b32\") " pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:54:05.186733 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.186707 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:05.197362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.197336 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zqlwh" Oct 30 08:54:05.271981 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.271907 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:05.498088 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.498013 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7888d9c8bc-nzrhq"] Oct 30 08:54:05.501020 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:54:05.500995 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf486f2e8_4167_421a_8f04_5f6ce10b6c25.slice/crio-f089a51a6979c8b20d73bb83ef34d5f7750798373e0740d8848539837c145efc WatchSource:0}: Error finding container f089a51a6979c8b20d73bb83ef34d5f7750798373e0740d8848539837c145efc: Status 404 returned error can't find the container with id f089a51a6979c8b20d73bb83ef34d5f7750798373e0740d8848539837c145efc Oct 30 08:54:05.507599 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:05.507580 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zqlwh"] Oct 30 08:54:05.512450 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:54:05.512423 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff1f68b_203c_43a3_b5a5_671eda399b32.slice/crio-e98484f2e543e5c05f5a18e94c47e8a01e6594cf9bfdd57276736c9f0adebdf2 WatchSource:0}: Error finding container e98484f2e543e5c05f5a18e94c47e8a01e6594cf9bfdd57276736c9f0adebdf2: Status 404 returned error can't find the container with id e98484f2e543e5c05f5a18e94c47e8a01e6594cf9bfdd57276736c9f0adebdf2 Oct 30 08:54:06.042798 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.042767 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:54:06.043241 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.043022 4920 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:54:06.441934 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.441912 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:54:06.485326 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.485297 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:06.496665 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.496629 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqlwh" event={"ID":"0ff1f68b-203c-43a3-b5a5-671eda399b32","Type":"ContainerStarted","Data":"d3d76350f9a318f76f55496bfa5e66fe21bc4356e883af7d7d38127faa6bbe8d"} Oct 30 08:54:06.497103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.496684 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqlwh" event={"ID":"0ff1f68b-203c-43a3-b5a5-671eda399b32","Type":"ContainerStarted","Data":"89b32b3fad206164a2aa5ee00853e156dfcf678aec8d2681015870cccfff79b1"} Oct 30 08:54:06.497103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.496708 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zqlwh" event={"ID":"0ff1f68b-203c-43a3-b5a5-671eda399b32","Type":"ContainerStarted","Data":"e98484f2e543e5c05f5a18e94c47e8a01e6594cf9bfdd57276736c9f0adebdf2"} Oct 30 08:54:06.498211 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.498181 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" event={"ID":"f486f2e8-4167-421a-8f04-5f6ce10b6c25","Type":"ContainerStarted","Data":"fb44856ddad73ef0d592814560e426d8f50fda4a3fa548362b5fb10324387303"} Oct 30 08:54:06.498340 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.498254 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" event={"ID":"f486f2e8-4167-421a-8f04-5f6ce10b6c25","Type":"ContainerStarted","Data":"f089a51a6979c8b20d73bb83ef34d5f7750798373e0740d8848539837c145efc"} Oct 30 08:54:06.677306 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.677174 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:06.874451 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:06.874418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:07.070935 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.070907 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:07.186832 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.186792 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:07.189517 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.189492 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:07.276861 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.276812 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:07.502206 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.502133 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:07.503419 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.503399 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7888d9c8bc-nzrhq" Oct 30 08:54:07.510099 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:07.510078 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86p9d" Oct 30 08:54:09.390268 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.390237 4920 scope.go:122] "RemoveContainer" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" Oct 30 08:54:09.511446 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.511421 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 08:54:09.511863 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.511846 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/2.log" Oct 30 08:54:09.511976 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.511883 4920 generic.go:356] "Generic (PLEG): container finished" podID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerID="0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b" exitCode=255 Oct 30 08:54:09.511976 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.511953 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerDied","Data":"0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b"} Oct 30 08:54:09.512080 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.512011 4920 scope.go:122] "RemoveContainer" containerID="83cf68e5cbc3302110b4bb7cd1a987be313d9eba768d939f021647acb105f957" Oct 30 08:54:09.512459 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:09.512440 4920 scope.go:122] "RemoveContainer" containerID="0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b" Oct 30 08:54:09.512623 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:09.512604 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:54:10.476299 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:10.476270 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mj9w5" Oct 30 08:54:10.516563 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:10.516538 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 08:54:11.592777 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.592745 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:54:11.593384 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.593166 4920 scope.go:122] "RemoveContainer" containerID="0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b" Oct 30 08:54:11.593447 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:11.593410 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:54:11.684973 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.684941 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:54:11.779810 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.779778 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:11.884019 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.883947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:11.979570 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:11.979535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:12.516859 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:12.516820 4920 patch_prober.go:29] interesting pod/image-registry-65ffb76754-ndfbg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 08:54:12.517059 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:12.516880 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:54:13.243910 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.243822 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:54:13.243910 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.243883 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:54:13.246233 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.246184 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d50e209-0bda-4891-be9d-9e3580efab4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-stsh2\" (UID: \"2d50e209-0bda-4891-be9d-9e3580efab4e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:54:13.246330 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.246240 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxh78\" (UID: \"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b\") " pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:54:13.286336 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.286302 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" Oct 30 08:54:13.300652 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.300623 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxh78" Oct 30 08:54:13.584022 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.583987 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2"] Oct 30 08:54:13.587653 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:54:13.587609 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d50e209_0bda_4891_be9d_9e3580efab4e.slice/crio-0f170fb89408f6729e0ff5a954958281b2631e767b34bdde89f48d3bd0a8e996 WatchSource:0}: Error finding container 0f170fb89408f6729e0ff5a954958281b2631e767b34bdde89f48d3bd0a8e996: Status 404 returned error can't find the container with id 0f170fb89408f6729e0ff5a954958281b2631e767b34bdde89f48d3bd0a8e996 Oct 30 08:54:13.619019 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.618990 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxh78"] Oct 30 08:54:13.621980 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:54:13.621945 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad7aaa6_1bb7_4e1e_bcf4_0e6e2744281b.slice/crio-e08e73b4acb43aa07f287f64e7d63c176b0d20018200cea3e12d1b128c98f503 WatchSource:0}: Error finding container e08e73b4acb43aa07f287f64e7d63c176b0d20018200cea3e12d1b128c98f503: Status 404 returned error can't find the container with id e08e73b4acb43aa07f287f64e7d63c176b0d20018200cea3e12d1b128c98f503 Oct 30 08:54:13.734656 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.734617 4920 patch_prober.go:29] interesting pod/image-registry-65ffb76754-ndfbg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 08:54:13.734826 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:13.734671 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:54:14.375445 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:14.375421 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-64jq8" Oct 30 08:54:14.533329 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:14.533295 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxh78" event={"ID":"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b","Type":"ContainerStarted","Data":"4a990ac8a01964cf8dfb032b3b485da6a40f45b6794542cc02bda7fec8fb4dc8"} Oct 30 08:54:14.533491 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:14.533359 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxh78" event={"ID":"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b","Type":"ContainerStarted","Data":"08cfb31cd44f2e8eaf3f75891e19c4971090c9f3ad4f3d8950e36d6686f00036"} Oct 30 08:54:14.533491 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:14.533383 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxh78" event={"ID":"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b","Type":"ContainerStarted","Data":"e08e73b4acb43aa07f287f64e7d63c176b0d20018200cea3e12d1b128c98f503"} Oct 30 08:54:14.534576 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:14.534553 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" event={"ID":"2d50e209-0bda-4891-be9d-9e3580efab4e","Type":"ContainerStarted","Data":"0f170fb89408f6729e0ff5a954958281b2631e767b34bdde89f48d3bd0a8e996"} Oct 30 08:54:15.548413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:15.548364 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-stsh2" event={"ID":"2d50e209-0bda-4891-be9d-9e3580efab4e","Type":"ContainerStarted","Data":"a422f97433047c897f89f309613a086435fd734968733b94429aac61575cc4f0"} Oct 30 08:54:15.887838 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:15.887753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:15.988044 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:15.988014 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:16.274659 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:16.274619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:16.430450 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:16.430419 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:16.553229 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:16.553186 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxh78" event={"ID":"4ad7aaa6-1bb7-4e1e-bcf4-0e6e2744281b","Type":"ContainerStarted","Data":"4dde0f08256657a0368c004663f3eef41f171f85cb8f407aa54f49055dbe2976"} Oct 30 08:54:16.854301 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:16.854207 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:16.950297 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:16.950260 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:17.447612 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:17.447551 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:17.530482 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:17.530434 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:18.371776 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:18.371745 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:18.473084 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:18.473052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:18.555462 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:18.555428 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:19.231230 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:19.231191 4920 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:54:19.231649 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:19.231631 4920 scope.go:122] "RemoveContainer" containerID="0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b" Oct 30 08:54:19.231859 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:19.231837 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mxzf_openshift-console-operator(adbbc457-38a1-45fd-ab3d-738f7dd07913)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" Oct 30 08:54:21.178920 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:21.178880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:21.463316 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:21.463199 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:21.690680 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:21.690648 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:54:22.792577 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:22.792524 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:22.982953 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:22.982914 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.093774 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.093693 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-frr44"] Oct 30 08:54:23.101976 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.101947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.117199 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.117174 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121099 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121072 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-textfile\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121113 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7sx\" (UniqueName: \"kubernetes.io/projected/c577300a-68a1-4849-99c2-d0af317d1432-kube-api-access-hg7sx\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121134 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-sys\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121159 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-tls\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121408 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121246 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121408 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121296 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-wtmp\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121408 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121342 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-metrics-client-ca\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121408 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121367 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-accelerators-collector-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.121642 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.121455 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-root\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.160389 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.160359 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 08:54:23.160389 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.160367 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jptkp\"" Oct 30 08:54:23.160797 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.160781 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 08:54:23.170797 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.170768 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 08:54:23.170797 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.170797 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 08:54:23.208022 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.207989 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.222130 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222096 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-textfile\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222149 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7sx\" (UniqueName: \"kubernetes.io/projected/c577300a-68a1-4849-99c2-d0af317d1432-kube-api-access-hg7sx\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222178 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-sys\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222210 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-tls\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222256 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222283 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-wtmp\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222296 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-sys\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222320 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-metrics-client-ca\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222342 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-accelerators-collector-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222681 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222368 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-root\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222681 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222444 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-root\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222681 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-wtmp\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.222681 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222508 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-textfile\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.223023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222983 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-metrics-client-ca\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.223023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.222983 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-accelerators-collector-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.224607 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.224588 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-tls\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.224691 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.224631 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c577300a-68a1-4849-99c2-d0af317d1432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.238709 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.238680 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7sx\" (UniqueName: \"kubernetes.io/projected/c577300a-68a1-4849-99c2-d0af317d1432-kube-api-access-hg7sx\") pod \"node-exporter-frr44\" (UID: \"c577300a-68a1-4849-99c2-d0af317d1432\") " pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.307868 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.307835 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.408988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.408909 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.469523 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.469488 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frr44" Oct 30 08:54:23.508312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.508256 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:23.509443 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:54:23.509416 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc577300a_68a1_4849_99c2_d0af317d1432.slice/crio-322d9f2ccc24adb1fc8f3f83af7455f77789c368ca9f3113bf317fce901db03b WatchSource:0}: Error finding container 322d9f2ccc24adb1fc8f3f83af7455f77789c368ca9f3113bf317fce901db03b: Status 404 returned error can't find the container with id 322d9f2ccc24adb1fc8f3f83af7455f77789c368ca9f3113bf317fce901db03b Oct 30 08:54:23.582396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.582358 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frr44" event={"ID":"c577300a-68a1-4849-99c2-d0af317d1432","Type":"ContainerStarted","Data":"322d9f2ccc24adb1fc8f3f83af7455f77789c368ca9f3113bf317fce901db03b"} Oct 30 08:54:23.734881 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.734796 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:54:23.986594 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:23.986506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:24.086040 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:24.086000 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:24.189072 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:24.189039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:24.341748 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:24.341711 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:24.442547 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:24.442514 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:25.592662 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:25.592621 4920 generic.go:356] "Generic (PLEG): container finished" podID="c577300a-68a1-4849-99c2-d0af317d1432" containerID="7d18413c64cf0909010cc6fbfc3a35f8e992e0289d33d700763ea8abae9aaec5" exitCode=0 Oct 30 08:54:25.593144 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:25.592667 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frr44" event={"ID":"c577300a-68a1-4849-99c2-d0af317d1432","Type":"ContainerDied","Data":"7d18413c64cf0909010cc6fbfc3a35f8e992e0289d33d700763ea8abae9aaec5"} Oct 30 08:54:26.296058 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.296026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:26.393142 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.393112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:26.445323 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.445283 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xb9sq" Oct 30 08:54:26.486097 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.486058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:26.598658 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.598582 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frr44" event={"ID":"c577300a-68a1-4849-99c2-d0af317d1432","Type":"ContainerStarted","Data":"c44c3da99df31abe77026d0e3d8eaa1a728a0bf4d5ed1c17038ae609826fb4b6"} Oct 30 08:54:26.599003 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.598659 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frr44" event={"ID":"c577300a-68a1-4849-99c2-d0af317d1432","Type":"ContainerStarted","Data":"dd58991b7eddb3fde4225e13742f4e7d0bcfbf4664e8de4ca3f0b0104b138fd9"} Oct 30 08:54:26.637090 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.637030 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-frr44" podStartSLOduration=2.191826302 podStartE2EDuration="3.637015351s" totalImagesPullingTime="1.366373849s" totalInitContainerRuntime="78.8152ms" isStatefulPod=true podCreationTimestamp="2026-10-30 08:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:26.636726127 +0000 UTC m=+30.360153270" watchObservedRunningTime="2026-10-30 08:54:26.637015351 +0000 UTC m=+30.360442492" Oct 30 08:54:26.746651 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.746615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:26.847192 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:26.847154 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:27.899298 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:27.899267 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:27.987741 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:27.987708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:29.848533 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:29.848500 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:29.940785 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:29.940749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:31.135291 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:31.135205 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:54:31.390202 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:31.390133 4920 scope.go:122] "RemoveContainer" containerID="0eec3343ea05b807b0e01b3a7799e271b7225ae864b4087c3b74c112ba98f40b" Oct 30 08:54:31.619670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:31.619639 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 08:54:31.619846 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:31.619693 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" event={"ID":"adbbc457-38a1-45fd-ab3d-738f7dd07913","Type":"ContainerStarted","Data":"b06ea9bcb941d095e8475faf09724a6bde48bd9a4a803e8458efb3dccd2fa2e3"} Oct 30 08:54:31.619963 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:31.619942 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:54:32.015265 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:32.015202 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" Oct 30 08:54:32.381983 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:32.381949 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:32.525724 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:32.525689 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:32.623259 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:32.623201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:32.816468 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:32.816434 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:33.021764 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:33.021732 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:33.122532 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:33.122459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:34.039347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.039312 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:54:34.135510 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.135474 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:34.310171 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.310090 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:34.410728 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.410689 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:34.536966 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.536933 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:34.631394 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:34.631312 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:36.705913 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:36.705878 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" podUID="25c4713f-3616-4d01-b7c7-310f70a14127" containerName="registry" containerID="cri-o://5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6" gracePeriod=30 Oct 30 08:54:36.938664 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:36.938635 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:54:37.030456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030379 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030432 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030456 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030732 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030500 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030732 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030533 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030732 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030558 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030732 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030585 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030732 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030631 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") pod \"25c4713f-3616-4d01-b7c7-310f70a14127\" (UID: \"25c4713f-3616-4d01-b7c7-310f70a14127\") " Oct 30 08:54:37.030994 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.030867 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:54:37.031385 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.031358 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:54:37.033444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.033406 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:54:37.033544 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.033488 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:37.033544 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.033517 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "kube-api-access-d6ljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:37.033544 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.033493 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:54:37.033655 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.033618 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:37.039406 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.039376 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted" pod "25c4713f-3616-4d01-b7c7-310f70a14127" (UID: "25c4713f-3616-4d01-b7c7-310f70a14127"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:54:37.131943 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131909 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ljr\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-kube-api-access-d6ljr\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.131943 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131939 4920 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-registry-certificates\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.131943 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131948 4920 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-installation-pull-secrets\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.132174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131958 4920 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25c4713f-3616-4d01-b7c7-310f70a14127-ca-trust-extracted\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.132174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131968 4920 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25c4713f-3616-4d01-b7c7-310f70a14127-image-registry-private-configuration\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.132174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131978 4920 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25c4713f-3616-4d01-b7c7-310f70a14127-trusted-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.132174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131986 4920 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-registry-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.132174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.131994 4920 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25c4713f-3616-4d01-b7c7-310f70a14127-bound-sa-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:37.143342 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.143308 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:37.242110 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.242073 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:37.345654 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.345580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:37.644568 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.644485 4920 generic.go:356] "Generic (PLEG): container finished" podID="25c4713f-3616-4d01-b7c7-310f70a14127" containerID="5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6" exitCode=0 Oct 30 08:54:37.644568 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.644529 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" Oct 30 08:54:37.644755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.644531 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" event={"ID":"25c4713f-3616-4d01-b7c7-310f70a14127","Type":"ContainerDied","Data":"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6"} Oct 30 08:54:37.644755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.644656 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd88cff75-jmrdx" event={"ID":"25c4713f-3616-4d01-b7c7-310f70a14127","Type":"ContainerDied","Data":"abb957b00b0f36212e8eddea2cc26c022dcabdec4a2b86795552ca59b5817980"} Oct 30 08:54:37.644755 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.644678 4920 scope.go:122] "RemoveContainer" containerID="5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6" Oct 30 08:54:37.653711 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.653694 4920 scope.go:122] "RemoveContainer" containerID="5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6" Oct 30 08:54:37.653979 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:37.653953 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6\": container with ID starting with 5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6 not found: ID does not exist" containerID="5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6" Oct 30 08:54:37.654079 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.653990 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6"} err="failed to get container status \"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6\": rpc error: code = NotFound desc = could not find container \"5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6\": container with ID starting with 5b091e193472b0d5ff210a11a8172f0251989b6c5af439770e4f00990bbd70b6 not found: ID does not exist" Oct 30 08:54:37.701810 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.701775 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:54:37.711639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:37.711608 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fd88cff75-jmrdx"] Oct 30 08:54:38.395812 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:38.395731 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c4713f-3616-4d01-b7c7-310f70a14127" path="/var/lib/kubelet/pods/25c4713f-3616-4d01-b7c7-310f70a14127/volumes" Oct 30 08:54:45.686769 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:45.686724 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:45.804554 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:45.804510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:47.451415 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:47.451372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:47.552025 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:47.551993 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:48.710394 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:48.710357 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:48.792993 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:48.792960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:59.062212 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.062175 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" containerID="cri-o://890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee" gracePeriod=30 Oct 30 08:54:59.305189 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.305166 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:54:59.416200 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416113 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416200 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416168 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416203 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416276 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416313 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416340 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416411 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416707 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416482 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") pod \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\" (UID: \"c86dd71b-65b4-48bc-9b09-772f6dd7053a\") " Oct 30 08:54:59.416707 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416639 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:54:59.416826 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416713 4920 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-trusted-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.416826 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.416747 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:54:59.421990 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.421943 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:59.423933 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.423801 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:54:59.423933 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.423869 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:59.424109 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.423913 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:54:59.424109 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.423965 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "kube-api-access-4pm8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:54:59.426169 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.426121 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted" pod "c86dd71b-65b4-48bc-9b09-772f6dd7053a" (UID: "c86dd71b-65b4-48bc-9b09-772f6dd7053a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:54:59.517647 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517611 4920 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-certificates\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517647 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517641 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pm8d\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-kube-api-access-4pm8d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517647 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517652 4920 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-image-registry-private-configuration\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517664 4920 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-bound-sa-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517673 4920 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86dd71b-65b4-48bc-9b09-772f6dd7053a-ca-trust-extracted\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517683 4920 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86dd71b-65b4-48bc-9b09-772f6dd7053a-registry-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.517849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.517692 4920 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86dd71b-65b4-48bc-9b09-772f6dd7053a-installation-pull-secrets\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:54:59.611690 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.611632 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:54:59.730456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.730377 4920 generic.go:356] "Generic (PLEG): container finished" podID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerID="890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee" exitCode=0 Oct 30 08:54:59.730456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.730422 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" event={"ID":"c86dd71b-65b4-48bc-9b09-772f6dd7053a","Type":"ContainerDied","Data":"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee"} Oct 30 08:54:59.730456 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.730435 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" Oct 30 08:54:59.730703 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.730490 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65ffb76754-ndfbg" event={"ID":"c86dd71b-65b4-48bc-9b09-772f6dd7053a","Type":"ContainerDied","Data":"4e0162dc7fa81fc0843630009593cb8ad773bb0ea1f4e220b1c6627d1b1f6819"} Oct 30 08:54:59.730703 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.730511 4920 scope.go:122] "RemoveContainer" containerID="890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee" Oct 30 08:54:59.738558 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.738536 4920 scope.go:122] "RemoveContainer" containerID="890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee" Oct 30 08:54:59.738802 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:54:59.738781 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee\": container with ID starting with 890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee not found: ID does not exist" containerID="890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee" Oct 30 08:54:59.738855 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.738814 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee"} err="failed to get container status \"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee\": rpc error: code = NotFound desc = could not find container \"890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee\": container with ID starting with 890185de8d17c745311805781ed2300524fce016a20dcb516b7e3fe90b7c9eee not found: ID does not exist" Oct 30 08:54:59.773032 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.773007 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:54:59.783089 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:54:59.783065 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65ffb76754-ndfbg"] Oct 30 08:55:00.394520 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:00.394482 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" path="/var/lib/kubelet/pods/c86dd71b-65b4-48bc-9b09-772f6dd7053a/volumes" Oct 30 08:55:00.508095 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:00.508062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:00.736514 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:00.736479 4920 generic.go:356] "Generic (PLEG): container finished" podID="8ae3e225-68cf-4814-8d33-e184ad008b3c" containerID="4a85237789a4bb96357bc56d3295526688f85c0c83a687c92275d942d9ea3771" exitCode=0 Oct 30 08:55:00.736669 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:00.736554 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" event={"ID":"8ae3e225-68cf-4814-8d33-e184ad008b3c","Type":"ContainerDied","Data":"4a85237789a4bb96357bc56d3295526688f85c0c83a687c92275d942d9ea3771"} Oct 30 08:55:00.736893 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:00.736876 4920 scope.go:122] "RemoveContainer" containerID="4a85237789a4bb96357bc56d3295526688f85c0c83a687c92275d942d9ea3771" Oct 30 08:55:01.133291 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:55:01.133256 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:55:01.742282 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:01.742247 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tpvfr" event={"ID":"8ae3e225-68cf-4814-8d33-e184ad008b3c","Type":"ContainerStarted","Data":"0854542dedfc625c949dcdd461448977a046d93b463a19db386ba1f54a227a5e"} Oct 30 08:55:01.936923 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:01.936887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:02.451030 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:02.450994 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:02.552082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:02.552047 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:08.823627 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:08.823589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:08.905200 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:08.905167 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:10.776452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:10.776413 4920 generic.go:356] "Generic (PLEG): container finished" podID="47f381ca-b571-4170-a89c-ecdef16c72f6" containerID="a563c6b875057f5e3bf261c6186f378e52117cd7b099a38d87de755dce51b83f" exitCode=0 Oct 30 08:55:10.776881 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:10.776486 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" event={"ID":"47f381ca-b571-4170-a89c-ecdef16c72f6","Type":"ContainerDied","Data":"a563c6b875057f5e3bf261c6186f378e52117cd7b099a38d87de755dce51b83f"} Oct 30 08:55:10.776881 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:10.776840 4920 scope.go:122] "RemoveContainer" containerID="a563c6b875057f5e3bf261c6186f378e52117cd7b099a38d87de755dce51b83f" Oct 30 08:55:11.162644 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:11.162609 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:11.782753 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:11.782720 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nnx9z" event={"ID":"47f381ca-b571-4170-a89c-ecdef16c72f6","Type":"ContainerStarted","Data":"3c0e6a9506698064897b219d9580090d8dd7a7b919a24603eecf5741cca40db5"} Oct 30 08:55:12.963737 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:12.963702 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:16.533439 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:16.533404 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:16.634083 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:16.634038 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:16.738787 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:16.738754 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:17.448821 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:17.448787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:17.536189 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:17.536157 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:18.111301 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:18.111254 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:18.192928 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:18.192893 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:20.752494 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:20.752421 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:20.855274 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:20.855237 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:25.838736 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:25.838657 4920 generic.go:356] "Generic (PLEG): container finished" podID="8f93bd2e-a21e-4bc1-993d-117adc818868" containerID="0d0935d2c6f63c57543fabf1561daa643bcf726e062c80674811e59463e5ceb5" exitCode=0 Oct 30 08:55:25.839098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:25.838728 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tn25f" event={"ID":"8f93bd2e-a21e-4bc1-993d-117adc818868","Type":"ContainerDied","Data":"0d0935d2c6f63c57543fabf1561daa643bcf726e062c80674811e59463e5ceb5"} Oct 30 08:55:25.839098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:25.839029 4920 scope.go:122] "RemoveContainer" containerID="0d0935d2c6f63c57543fabf1561daa643bcf726e062c80674811e59463e5ceb5" Oct 30 08:55:26.844699 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:26.844661 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tn25f" event={"ID":"8f93bd2e-a21e-4bc1-993d-117adc818868","Type":"ContainerStarted","Data":"6c1964a15f81cd1caf3ec778f4e8427881a0a25b96187539aec284905d5567d3"} Oct 30 08:55:31.133657 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:55:31.133619 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:55:32.458863 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:32.458829 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:32.559151 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:32.559116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:32.642817 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:32.642783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:55:45.792198 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:55:45.792151 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:56:01.134381 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:56:01.134335 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:56:31.134621 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:56:31.134581 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:56:43.378187 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:56:43.378146 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:56:45.908739 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:56:45.908698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:01.133854 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:57:01.133809 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:57:03.115202 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:03.115160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:03.207243 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:03.207188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:26.536427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:26.536387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:26.618852 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:26.618820 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:31.133924 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:57:31.133881 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:57:38.028799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:38.028765 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:41.595136 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:41.595042 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:41.711063 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:41.711031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:41.847281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:41.847192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:41.974625 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:41.974585 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:42.547486 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547452 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8"] Oct 30 08:57:42.547757 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547745 4920 cpu_manager.go:414] "containerMap: removing container" podUID="25c4713f-3616-4d01-b7c7-310f70a14127" containerName="registry" Oct 30 08:57:42.547802 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547757 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c4713f-3616-4d01-b7c7-310f70a14127" containerName="registry" Oct 30 08:57:42.547802 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547767 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" Oct 30 08:57:42.547802 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547772 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" Oct 30 08:57:42.547915 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547822 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86dd71b-65b4-48bc-9b09-772f6dd7053a" containerName="registry" Oct 30 08:57:42.547915 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.547835 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c4713f-3616-4d01-b7c7-310f70a14127" containerName="registry" Oct 30 08:57:42.550991 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.550970 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.578253 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.578208 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-2rzfd\"" Oct 30 08:57:42.578398 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.578288 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 08:57:42.578398 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.578342 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 08:57:42.579665 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.579634 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8"] Oct 30 08:57:42.626051 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.626014 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78rs\" (UniqueName: \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.626051 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.626051 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.626497 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.626079 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.626788 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.626765 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6"] Oct 30 08:57:42.630065 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.630048 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.651736 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.651708 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6"] Oct 30 08:57:42.667757 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.667732 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qtqh7\"" Oct 30 08:57:42.727043 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727001 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.727263 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727071 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.727263 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.727263 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78rs\" (UniqueName: \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.727427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727284 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8xl\" (UniqueName: \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.727427 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727355 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.727527 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727469 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.727689 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.727669 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.746264 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.746230 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78rs\" (UniqueName: \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.828672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.828589 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.828672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.828633 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.828849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.828749 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8xl\" (UniqueName: \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.833645 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.829264 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.833645 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.829280 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.846063 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.846034 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8xl\" (UniqueName: \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.866666 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.866634 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:57:42.945974 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.945943 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:57:42.968589 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:42.968553 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:43.068596 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.068558 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:43.160115 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.160032 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8"] Oct 30 08:57:43.165096 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:43.165067 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70557c79_8cc6_4877_9282_3f950e2a4f2c.slice/crio-de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c WatchSource:0}: Error finding container de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c: Status 404 returned error can't find the container with id de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c Oct 30 08:57:43.250018 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.249971 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6"] Oct 30 08:57:43.254727 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:43.254694 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021afb5a_edd8_4c17_9aaf_f4a1e45b9591.slice/crio-ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44 WatchSource:0}: Error finding container ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44: Status 404 returned error can't find the container with id ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44 Oct 30 08:57:43.340516 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.340481 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" event={"ID":"021afb5a-edd8-4c17-9aaf-f4a1e45b9591","Type":"ContainerStarted","Data":"ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44"} Oct 30 08:57:43.341602 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.341575 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" event={"ID":"70557c79-8cc6-4877-9282-3f950e2a4f2c","Type":"ContainerStarted","Data":"de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c"} Oct 30 08:57:43.883383 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:43.883322 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:46.019672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.019630 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:46.328556 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.327662 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9"] Oct 30 08:57:46.332429 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.332401 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.356550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.356519 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-xblws\"" Oct 30 08:57:46.359620 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.358543 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9"] Oct 30 08:57:46.426575 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.426541 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb"] Oct 30 08:57:46.430594 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.430565 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.446836 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.446800 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gppxq\"" Oct 30 08:57:46.447758 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.447724 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb"] Oct 30 08:57:46.461483 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.461443 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.461672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.461492 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.461672 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.461628 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwm7\" (UniqueName: \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.524980 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.524945 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw"] Oct 30 08:57:46.529622 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.529568 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt"] Oct 30 08:57:46.529791 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.529675 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.533554 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.533529 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.546848 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.546817 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw"] Oct 30 08:57:46.549023 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.548253 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-zbw6j\"" Oct 30 08:57:46.549942 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.549750 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vfjmt\"" Oct 30 08:57:46.550289 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.550267 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt"] Oct 30 08:57:46.562146 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562115 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwm7\" (UniqueName: \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.562415 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562391 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.562522 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562424 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5nd\" (UniqueName: \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.562522 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562456 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.562648 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562579 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.562648 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562639 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.562880 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562861 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.562880 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.562871 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.580037 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.579939 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwm7\" (UniqueName: \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.622396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.622015 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk"] Oct 30 08:57:46.627154 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.627122 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.644388 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.644359 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk"] Oct 30 08:57:46.648453 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.648419 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pk9km\"" Oct 30 08:57:46.651667 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.651469 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:57:46.664824 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.664792 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmv5d\" (UniqueName: \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.664998 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.664911 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.665075 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665007 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.665075 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665043 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.665174 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665107 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.665251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665182 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.665251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665211 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzzr\" (UniqueName: \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.665362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665269 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5nd\" (UniqueName: \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.665362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665296 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.665362 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665331 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.665625 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665364 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscgt\" (UniqueName: \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.665625 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665388 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.665818 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.665795 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.666079 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.666060 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.683850 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.683813 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5nd\" (UniqueName: \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.753354 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.753043 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:57:46.765989 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.765964 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.766098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766007 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.766098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766044 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzzr\" (UniqueName: \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.766098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766066 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.766098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766086 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.766324 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766106 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscgt\" (UniqueName: \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.766324 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766121 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.766324 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766142 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmv5d\" (UniqueName: \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.766324 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766160 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.766522 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766437 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.766522 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766477 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.766747 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766715 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.766747 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.766738 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.767030 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.767001 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.767199 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.767163 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.784633 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.784583 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscgt\" (UniqueName: \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.785344 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.785298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmv5d\" (UniqueName: \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.785487 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.785466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzzr\" (UniqueName: \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.852153 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.852075 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:57:46.861152 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.860964 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:57:46.947914 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.947662 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:57:46.973344 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:46.972921 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9"] Oct 30 08:57:46.983969 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:46.983931 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87367b4c_77d5_42d1_bcd6_29da099871f8.slice/crio-e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97 WatchSource:0}: Error finding container e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97: Status 404 returned error can't find the container with id e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97 Oct 30 08:57:47.041411 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.041376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:47.071086 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:47.071052 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d069b2_6c32_49b8_8374_a6122355f4d3.slice/crio-55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429 WatchSource:0}: Error finding container 55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429: Status 404 returned error can't find the container with id 55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429 Oct 30 08:57:47.071981 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.071958 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb"] Oct 30 08:57:47.138372 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.138291 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:47.157380 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.157345 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw"] Oct 30 08:57:47.162884 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:47.162850 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbfab17_a53f_4b89_be97_ea6fab7da2e7.slice/crio-13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5 WatchSource:0}: Error finding container 13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5: Status 404 returned error can't find the container with id 13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5 Oct 30 08:57:47.201258 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.201196 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt"] Oct 30 08:57:47.206263 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:47.206205 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96030f5_5dc2_41cc_bf90_5fe5c6dd2abb.slice/crio-53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595 WatchSource:0}: Error finding container 53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595: Status 404 returned error can't find the container with id 53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595 Oct 30 08:57:47.263829 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.263791 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk"] Oct 30 08:57:47.268679 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:47.268645 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfa41f7_1d9b_4334_9b60_dafb65f462dd.slice/crio-c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c WatchSource:0}: Error finding container c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c: Status 404 returned error can't find the container with id c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c Oct 30 08:57:47.366114 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.366072 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" event={"ID":"ebfa41f7-1d9b-4334-9b60-dafb65f462dd","Type":"ContainerStarted","Data":"c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c"} Oct 30 08:57:47.367680 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.367645 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" event={"ID":"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb","Type":"ContainerStarted","Data":"53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595"} Oct 30 08:57:47.369553 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.369528 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerStarted","Data":"e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97"} Oct 30 08:57:47.371499 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.371438 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" event={"ID":"09d069b2-6c32-49b8-8374-a6122355f4d3","Type":"ContainerStarted","Data":"55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429"} Oct 30 08:57:47.373046 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.373020 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" event={"ID":"5fbfab17-a53f-4b89-be97-ea6fab7da2e7","Type":"ContainerStarted","Data":"13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5"} Oct 30 08:57:47.727571 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.727540 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s"] Oct 30 08:57:47.730942 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.730914 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.745251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.743696 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7g5p\"" Oct 30 08:57:47.745251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.743870 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 08:57:47.757146 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.757114 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s"] Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776755 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776805 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtqlf\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-kube-api-access-mtqlf\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776844 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776884 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776908 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776938 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.776978 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.777016 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.777273 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.777042 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.789347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.789209 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:47.859799 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.859477 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55"] Oct 30 08:57:47.865454 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.864878 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.880671 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.880710 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtqlf\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-kube-api-access-mtqlf\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881076 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881125 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881171 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881198 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881312 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881373 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881411 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.881550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881439 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.882158 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881719 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-nnm78\"" Oct 30 08:57:47.882158 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881786 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.882158 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.881928 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.882916 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.882812 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.883350 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.883324 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.888603 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.886035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.888603 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.886780 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55"] Oct 30 08:57:47.888603 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.887395 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.913744 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.913711 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.914138 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.914109 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtqlf\" (UniqueName: \"kubernetes.io/projected/d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c-kube-api-access-mtqlf\") pod \"maas-default-gateway-openshift-default-77f556d647-z896s\" (UID: \"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:47.930010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.929925 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:47.982851 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982766 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.982851 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982824 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983036 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982868 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983036 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982905 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983036 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982955 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983036 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.982983 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5653d03d-10e3-41a3-81ec-275974428909-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983036 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.983014 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983185 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.983044 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448xz\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-kube-api-access-448xz\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:47.983185 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:47.983105 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5653d03d-10e3-41a3-81ec-275974428909-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.033470 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.033440 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:48.048303 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.048051 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.083771 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5653d03d-10e3-41a3-81ec-275974428909-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.083837 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.083874 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.083928 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.083970 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.084025 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.084052 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5653d03d-10e3-41a3-81ec-275974428909-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.084085 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.084111 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448xz\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-kube-api-access-448xz\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.085488 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.085424 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.086082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.085638 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5653d03d-10e3-41a3-81ec-275974428909-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.086082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.085887 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.086082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.085903 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.086082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.086043 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.090251 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.090209 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5653d03d-10e3-41a3-81ec-275974428909-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.106683 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.106632 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5653d03d-10e3-41a3-81ec-275974428909-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.107464 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.107414 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.112431 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.112403 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-448xz\" (UniqueName: \"kubernetes.io/projected/5653d03d-10e3-41a3-81ec-275974428909-kube-api-access-448xz\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55\" (UID: \"5653d03d-10e3-41a3-81ec-275974428909\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.183939 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.183650 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:48.421234 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.421186 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s"] Oct 30 08:57:48.426471 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:48.426416 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-24f136c20970024916696b0fc8230bcd83552bf586c919e35e3d63fe7aef218c WatchSource:0}: Error finding container 24f136c20970024916696b0fc8230bcd83552bf586c919e35e3d63fe7aef218c: Status 404 returned error can't find the container with id 24f136c20970024916696b0fc8230bcd83552bf586c919e35e3d63fe7aef218c Oct 30 08:57:48.511286 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:48.511186 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55"] Oct 30 08:57:48.516077 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:48.516040 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5653d03d_10e3_41a3_81ec_275974428909.slice/crio-f5e6f57a7611185ece0bfb66a5ad984f322913427b48b898f4a76f13917b0f6f WatchSource:0}: Error finding container f5e6f57a7611185ece0bfb66a5ad984f322913427b48b898f4a76f13917b0f6f: Status 404 returned error can't find the container with id f5e6f57a7611185ece0bfb66a5ad984f322913427b48b898f4a76f13917b0f6f Oct 30 08:57:49.403185 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:49.403097 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" event={"ID":"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c","Type":"ContainerStarted","Data":"24f136c20970024916696b0fc8230bcd83552bf586c919e35e3d63fe7aef218c"} Oct 30 08:57:49.436183 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:49.436077 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" event={"ID":"5653d03d-10e3-41a3-81ec-275974428909","Type":"ContainerStarted","Data":"f5e6f57a7611185ece0bfb66a5ad984f322913427b48b898f4a76f13917b0f6f"} Oct 30 08:57:50.621310 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.619425 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227"] Oct 30 08:57:50.625049 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.624451 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.666473 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.666437 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227"] Oct 30 08:57:50.715580 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.715277 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.715580 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.715364 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgsj\" (UniqueName: \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.715580 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.715460 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.818050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.816939 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.818050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.816994 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.818050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.817063 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgsj\" (UniqueName: \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.818050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.817769 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.818050 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.818012 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.841481 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.841384 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgsj\" (UniqueName: \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:50.895723 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.895523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:57:50.956575 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:50.956304 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:57:51.344513 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:51.344475 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227"] Oct 30 08:57:51.367048 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:57:51.367007 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2754164_3d8d_4ab6_a9ea_5bc0af97ae77.slice/crio-a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8 WatchSource:0}: Error finding container a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8: Status 404 returned error can't find the container with id a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8 Oct 30 08:57:51.457783 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:51.457742 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" event={"ID":"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77","Type":"ContainerStarted","Data":"a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8"} Oct 30 08:57:55.510880 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:55.510798 4920 generic.go:356] "Generic (PLEG): container finished" podID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerID="f3473dc91971943e790a055b6ace2b6d5302fe196e0e8cf7d130bc77cff20a60" exitCode=0 Oct 30 08:57:55.511358 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:55.510935 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" event={"ID":"70557c79-8cc6-4877-9282-3f950e2a4f2c","Type":"ContainerDied","Data":"f3473dc91971943e790a055b6ace2b6d5302fe196e0e8cf7d130bc77cff20a60"} Oct 30 08:57:55.520308 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:55.519852 4920 generic.go:356] "Generic (PLEG): container finished" podID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerID="4e99dbefe4fe8ae6f603ae8be902b81884025add18001ceb355bc0caf09a91e3" exitCode=0 Oct 30 08:57:55.520308 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:55.519904 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" event={"ID":"021afb5a-edd8-4c17-9aaf-f4a1e45b9591","Type":"ContainerDied","Data":"4e99dbefe4fe8ae6f603ae8be902b81884025add18001ceb355bc0caf09a91e3"} Oct 30 08:57:56.936175 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:56.935401 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:56.936175 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:56.935486 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:56.936175 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:56.935528 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:57.021791 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.021749 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:57.021950 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.021834 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:57.021950 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.021876 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:57.541508 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.541333 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" event={"ID":"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c","Type":"ContainerStarted","Data":"83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b"} Oct 30 08:57:57.551770 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.550967 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerStarted","Data":"151c1b78e72bf15276cfc34b1cb3d78702c5e061ac3963f471b5d341e67f3a23"} Oct 30 08:57:57.557082 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.556474 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" event={"ID":"5653d03d-10e3-41a3-81ec-275974428909","Type":"ContainerStarted","Data":"b8c2372c311d25d2ad92addf540f02c9bf29e0f8cadcb8aea652380ab6e74e9c"} Oct 30 08:57:57.582953 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.582682 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podStartSLOduration=1.990550348 podStartE2EDuration="10.58266211s" totalImagesPullingTime="8.592111762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:57:57.580226751 +0000 UTC m=+241.303653915" watchObservedRunningTime="2026-10-30 08:57:57.58266211 +0000 UTC m=+241.306089255" Oct 30 08:57:57.619379 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:57.618956 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" podStartSLOduration=2.20713385 podStartE2EDuration="10.618936572s" totalImagesPullingTime="8.411802722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:57:57.615989119 +0000 UTC m=+241.339416266" watchObservedRunningTime="2026-10-30 08:57:57.618936572 +0000 UTC m=+241.342363719" Oct 30 08:57:58.048958 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.048918 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:58.054921 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.054635 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:58.184416 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.184322 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:58.189890 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.189860 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:58.564371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.564334 4920 generic.go:356] "Generic (PLEG): container finished" podID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerID="e292e0057d15c0f91fa65932c5c7438cd216d9eed72a05a5bf7f24a2de31450e" exitCode=0 Oct 30 08:57:58.564527 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.564426 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" event={"ID":"5fbfab17-a53f-4b89-be97-ea6fab7da2e7","Type":"ContainerDied","Data":"e292e0057d15c0f91fa65932c5c7438cd216d9eed72a05a5bf7f24a2de31450e"} Oct 30 08:57:58.570620 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.567756 4920 generic.go:356] "Generic (PLEG): container finished" podID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerID="9b49cb5e9ec0be5b40b7900c74a97400094a477240d4e9266ba4871883dd58d8" exitCode=0 Oct 30 08:57:58.570620 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.567862 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" event={"ID":"ebfa41f7-1d9b-4334-9b60-dafb65f462dd","Type":"ContainerDied","Data":"9b49cb5e9ec0be5b40b7900c74a97400094a477240d4e9266ba4871883dd58d8"} Oct 30 08:57:58.573379 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.573342 4920 generic.go:356] "Generic (PLEG): container finished" podID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerID="8c6b593c300c1a218e1112b57bd88603a18b55467c33125d99a23114b22f0c69" exitCode=0 Oct 30 08:57:58.573489 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.573440 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" event={"ID":"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb","Type":"ContainerDied","Data":"8c6b593c300c1a218e1112b57bd88603a18b55467c33125d99a23114b22f0c69"} Oct 30 08:57:58.577546 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.577520 4920 generic.go:356] "Generic (PLEG): container finished" podID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerID="151c1b78e72bf15276cfc34b1cb3d78702c5e061ac3963f471b5d341e67f3a23" exitCode=0 Oct 30 08:57:58.577633 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.577549 4920 generic.go:356] "Generic (PLEG): container finished" podID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerID="e983514a29ba9ed723f30581027ddd47b28b29e7f54f78f6c25624edc2acfc40" exitCode=0 Oct 30 08:57:58.577704 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.577638 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerDied","Data":"151c1b78e72bf15276cfc34b1cb3d78702c5e061ac3963f471b5d341e67f3a23"} Oct 30 08:57:58.577704 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.577675 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerDied","Data":"e983514a29ba9ed723f30581027ddd47b28b29e7f54f78f6c25624edc2acfc40"} Oct 30 08:57:58.581492 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.580167 4920 generic.go:356] "Generic (PLEG): container finished" podID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerID="95b564cfcf0eb46c9242099efdac0e9404450541761b7e2468c88638c592a5c3" exitCode=0 Oct 30 08:57:58.582903 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.581579 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" event={"ID":"09d069b2-6c32-49b8-8374-a6122355f4d3","Type":"ContainerDied","Data":"95b564cfcf0eb46c9242099efdac0e9404450541761b7e2468c88638c592a5c3"} Oct 30 08:57:58.582903 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.581630 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:58.582903 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.582398 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:58.583479 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.583459 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55" Oct 30 08:57:58.583842 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:58.583824 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 08:57:59.591105 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.591079 4920 generic.go:356] "Generic (PLEG): container finished" podID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerID="913f92a0e4e5ec5fefdc4ab1bcd317334b0181649d89c4507ea94ac89a97ac28" exitCode=0 Oct 30 08:57:59.591503 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.591206 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" event={"ID":"09d069b2-6c32-49b8-8374-a6122355f4d3","Type":"ContainerDied","Data":"913f92a0e4e5ec5fefdc4ab1bcd317334b0181649d89c4507ea94ac89a97ac28"} Oct 30 08:57:59.597128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.596435 4920 generic.go:356] "Generic (PLEG): container finished" podID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerID="80ef66d28bbeb5012d3b2551ea0ca291c56b246102f52d2840323a96f7400ae3" exitCode=0 Oct 30 08:57:59.597128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.596551 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" event={"ID":"5fbfab17-a53f-4b89-be97-ea6fab7da2e7","Type":"ContainerDied","Data":"80ef66d28bbeb5012d3b2551ea0ca291c56b246102f52d2840323a96f7400ae3"} Oct 30 08:57:59.602642 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.602606 4920 generic.go:356] "Generic (PLEG): container finished" podID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerID="2b73248e61962540077191baa6b7e15d4565c88a03954b8cfd540b24cba3ee23" exitCode=0 Oct 30 08:57:59.602758 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.602700 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" event={"ID":"ebfa41f7-1d9b-4334-9b60-dafb65f462dd","Type":"ContainerDied","Data":"2b73248e61962540077191baa6b7e15d4565c88a03954b8cfd540b24cba3ee23"} Oct 30 08:57:59.608540 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.606876 4920 generic.go:356] "Generic (PLEG): container finished" podID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerID="6cece67a99d37a371424cc975e2736470d85f8b97be838fa90c7da48334a8587" exitCode=0 Oct 30 08:57:59.608540 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:57:59.607986 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" event={"ID":"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb","Type":"ContainerDied","Data":"6cece67a99d37a371424cc975e2736470d85f8b97be838fa90c7da48334a8587"} Oct 30 08:58:00.616639 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.616598 4920 generic.go:356] "Generic (PLEG): container finished" podID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerID="dd5f62f336f10285024146f6252023ec2ee11cc085e0173b5767e25280ca2c9d" exitCode=0 Oct 30 08:58:00.617106 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.616709 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" event={"ID":"70557c79-8cc6-4877-9282-3f950e2a4f2c","Type":"ContainerDied","Data":"dd5f62f336f10285024146f6252023ec2ee11cc085e0173b5767e25280ca2c9d"} Oct 30 08:58:00.621941 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.621914 4920 generic.go:356] "Generic (PLEG): container finished" podID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerID="455099e1ffa43ad6ab9bee11c9024b46fd316aad5393440442e9cebe4ad2e081" exitCode=0 Oct 30 08:58:00.622048 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.622010 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" event={"ID":"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77","Type":"ContainerDied","Data":"455099e1ffa43ad6ab9bee11c9024b46fd316aad5393440442e9cebe4ad2e081"} Oct 30 08:58:00.629684 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.628661 4920 generic.go:356] "Generic (PLEG): container finished" podID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerID="4e54a0181aa3c60cce09e1667eff5209976bb6fe53b7f9238eeb3c2cc3beda8d" exitCode=0 Oct 30 08:58:00.629684 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:00.628764 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" event={"ID":"021afb5a-edd8-4c17-9aaf-f4a1e45b9591","Type":"ContainerDied","Data":"4e54a0181aa3c60cce09e1667eff5209976bb6fe53b7f9238eeb3c2cc3beda8d"} Oct 30 08:58:01.159284 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:58:01.159208 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:58:01.641135 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:01.641091 4920 generic.go:356] "Generic (PLEG): container finished" podID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerID="1028e74c5d94d0a9dd5aaf2a8492ce4018ad67892f12be54be1f3de7e4ce8411" exitCode=0 Oct 30 08:58:01.641664 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:01.641198 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" event={"ID":"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77","Type":"ContainerDied","Data":"1028e74c5d94d0a9dd5aaf2a8492ce4018ad67892f12be54be1f3de7e4ce8411"} Oct 30 08:58:07.792052 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:07.791959 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:09.380153 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:09.380052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:09.477874 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:09.477840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:14.779202 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:14.779122 4920 generic.go:356] "Generic (PLEG): container finished" podID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerID="667f9415aa39f2ca2f7f862480646e2347700f95720931e165b2d2cd05b2fc09" exitCode=0 Oct 30 08:58:14.779707 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:14.779260 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerDied","Data":"667f9415aa39f2ca2f7f862480646e2347700f95720931e165b2d2cd05b2fc09"} Oct 30 08:58:15.980962 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:15.980931 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:58:16.127086 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.127058 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\") pod \"87367b4c-77d5-42d1-bcd6-29da099871f8\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " Oct 30 08:58:16.127205 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.127174 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\") pod \"87367b4c-77d5-42d1-bcd6-29da099871f8\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " Oct 30 08:58:16.127290 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.127240 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\" (UniqueName: \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\") pod \"87367b4c-77d5-42d1-bcd6-29da099871f8\" (UID: \"87367b4c-77d5-42d1-bcd6-29da099871f8\") " Oct 30 08:58:16.128176 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.128144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle" pod "87367b4c-77d5-42d1-bcd6-29da099871f8" (UID: "87367b4c-77d5-42d1-bcd6-29da099871f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:16.128313 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.128290 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:16.130907 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.130874 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7" pod "87367b4c-77d5-42d1-bcd6-29da099871f8" (UID: "87367b4c-77d5-42d1-bcd6-29da099871f8"). InnerVolumeSpecName "kube-api-access-fhwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:16.136511 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.136485 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util" pod "87367b4c-77d5-42d1-bcd6-29da099871f8" (UID: "87367b4c-77d5-42d1-bcd6-29da099871f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:16.229705 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.229672 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhwm7\" (UniqueName: \"kubernetes.io/projected/87367b4c-77d5-42d1-bcd6-29da099871f8-kube-api-access-fhwm7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:16.229826 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.229709 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87367b4c-77d5-42d1-bcd6-29da099871f8-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:16.793261 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.793206 4920 generic.go:356] "Generic (PLEG): container finished" podID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerID="be80cd670cb3e678aab9720edf28c7afbd195bdcf746ee6f041d3f4831f64f7d" exitCode=0 Oct 30 08:58:16.793483 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.793351 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" event={"ID":"09d069b2-6c32-49b8-8374-a6122355f4d3","Type":"ContainerDied","Data":"be80cd670cb3e678aab9720edf28c7afbd195bdcf746ee6f041d3f4831f64f7d"} Oct 30 08:58:16.796409 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.796374 4920 generic.go:356] "Generic (PLEG): container finished" podID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerID="6894062197ec91e298247aad81d993093a87c2e3a3e1b01f07961845708a1d8b" exitCode=0 Oct 30 08:58:16.796550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.796509 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" event={"ID":"5fbfab17-a53f-4b89-be97-ea6fab7da2e7","Type":"ContainerDied","Data":"6894062197ec91e298247aad81d993093a87c2e3a3e1b01f07961845708a1d8b"} Oct 30 08:58:16.799652 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.799620 4920 generic.go:356] "Generic (PLEG): container finished" podID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerID="8001f4eedd488acc09378f6ece51594724b4c27586f46fd3346f488a73480f84" exitCode=0 Oct 30 08:58:16.799760 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.799707 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" event={"ID":"ebfa41f7-1d9b-4334-9b60-dafb65f462dd","Type":"ContainerDied","Data":"8001f4eedd488acc09378f6ece51594724b4c27586f46fd3346f488a73480f84"} Oct 30 08:58:16.802437 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.802408 4920 generic.go:356] "Generic (PLEG): container finished" podID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerID="31c17973c29dc7ae86dadc7557b9e85b9bbd627e9451a5b8a60d326cb2fc8ab5" exitCode=0 Oct 30 08:58:16.802563 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.802505 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" event={"ID":"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb","Type":"ContainerDied","Data":"31c17973c29dc7ae86dadc7557b9e85b9bbd627e9451a5b8a60d326cb2fc8ab5"} Oct 30 08:58:16.804897 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.804872 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" event={"ID":"87367b4c-77d5-42d1-bcd6-29da099871f8","Type":"ContainerDied","Data":"e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97"} Oct 30 08:58:16.804999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.804917 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e518881bfac6c1cfed325738d178457c050ead0b915416316c4034d745af97" Oct 30 08:58:16.804999 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:16.804884 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9s8z9" Oct 30 08:58:17.817641 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.817607 4920 generic.go:356] "Generic (PLEG): container finished" podID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerID="e73fa15f3b73df02a3ca317edb51bfbe8e8f4963479daa6c87d425da9a0b0186" exitCode=0 Oct 30 08:58:17.818070 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.817670 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" event={"ID":"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77","Type":"ContainerDied","Data":"e73fa15f3b73df02a3ca317edb51bfbe8e8f4963479daa6c87d425da9a0b0186"} Oct 30 08:58:17.819701 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.819677 4920 generic.go:356] "Generic (PLEG): container finished" podID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerID="beb61023f9c65d8ffda2bb2fa0fa8342f3ca181c6f75fd97cf6e1106efcb6e65" exitCode=0 Oct 30 08:58:17.819826 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.819758 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" event={"ID":"021afb5a-edd8-4c17-9aaf-f4a1e45b9591","Type":"ContainerDied","Data":"beb61023f9c65d8ffda2bb2fa0fa8342f3ca181c6f75fd97cf6e1106efcb6e65"} Oct 30 08:58:17.821632 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.821614 4920 generic.go:356] "Generic (PLEG): container finished" podID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerID="d8dafe7e926c64ea69b23b933d93ded2a720919b43fea49e66389b442bac7009" exitCode=0 Oct 30 08:58:17.821704 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.821681 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" event={"ID":"70557c79-8cc6-4877-9282-3f950e2a4f2c","Type":"ContainerDied","Data":"d8dafe7e926c64ea69b23b933d93ded2a720919b43fea49e66389b442bac7009"} Oct 30 08:58:17.983303 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:17.983232 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:58:18.047297 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.045812 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\" (UniqueName: \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\") pod \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " Oct 30 08:58:18.047297 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.045953 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\") pod \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " Oct 30 08:58:18.047297 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.046019 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\") pod \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\" (UID: \"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb\") " Oct 30 08:58:18.047576 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.047466 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle" pod "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" (UID: "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.051009 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.050980 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt" pod "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" (UID: "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb"). InnerVolumeSpecName "kube-api-access-hscgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:18.052692 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.052666 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util" pod "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" (UID: "a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.055694 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.055673 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:58:18.058035 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.058016 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:58:18.061398 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.061381 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:58:18.147187 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147157 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\") pod \"09d069b2-6c32-49b8-8374-a6122355f4d3\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " Oct 30 08:58:18.147403 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147235 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\" (UniqueName: \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\") pod \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " Oct 30 08:58:18.147403 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147295 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\" (UniqueName: \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\") pod \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " Oct 30 08:58:18.147403 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147326 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\") pod \"09d069b2-6c32-49b8-8374-a6122355f4d3\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " Oct 30 08:58:18.147403 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147352 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\") pod \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " Oct 30 08:58:18.147617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147467 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\") pod \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " Oct 30 08:58:18.147617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147554 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\") pod \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\" (UID: \"ebfa41f7-1d9b-4334-9b60-dafb65f462dd\") " Oct 30 08:58:18.147617 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147604 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\" (UniqueName: \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\") pod \"09d069b2-6c32-49b8-8374-a6122355f4d3\" (UID: \"09d069b2-6c32-49b8-8374-a6122355f4d3\") " Oct 30 08:58:18.147759 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147652 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\") pod \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\" (UID: \"5fbfab17-a53f-4b89-be97-ea6fab7da2e7\") " Oct 30 08:58:18.147847 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.147816 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle" pod "09d069b2-6c32-49b8-8374-a6122355f4d3" (UID: "09d069b2-6c32-49b8-8374-a6122355f4d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.148055 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.148035 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscgt\" (UniqueName: \"kubernetes.io/projected/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-kube-api-access-hscgt\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.148103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.148061 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.148103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.148077 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.148103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.148089 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.148556 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.148524 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle" pod "ebfa41f7-1d9b-4334-9b60-dafb65f462dd" (UID: "ebfa41f7-1d9b-4334-9b60-dafb65f462dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.149977 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.149879 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle" pod "5fbfab17-a53f-4b89-be97-ea6fab7da2e7" (UID: "5fbfab17-a53f-4b89-be97-ea6fab7da2e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.150274 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.150248 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr" pod "ebfa41f7-1d9b-4334-9b60-dafb65f462dd" (UID: "ebfa41f7-1d9b-4334-9b60-dafb65f462dd"). InnerVolumeSpecName "kube-api-access-mhzzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:18.150472 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.150441 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd" pod "09d069b2-6c32-49b8-8374-a6122355f4d3" (UID: "09d069b2-6c32-49b8-8374-a6122355f4d3"). InnerVolumeSpecName "kube-api-access-cb5nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:18.150595 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.150573 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d" pod "5fbfab17-a53f-4b89-be97-ea6fab7da2e7" (UID: "5fbfab17-a53f-4b89-be97-ea6fab7da2e7"). InnerVolumeSpecName "kube-api-access-qmv5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:18.153404 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.153373 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util" pod "ebfa41f7-1d9b-4334-9b60-dafb65f462dd" (UID: "ebfa41f7-1d9b-4334-9b60-dafb65f462dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.153895 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.153873 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util" pod "09d069b2-6c32-49b8-8374-a6122355f4d3" (UID: "09d069b2-6c32-49b8-8374-a6122355f4d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.154131 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.154111 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util" pod "5fbfab17-a53f-4b89-be97-ea6fab7da2e7" (UID: "5fbfab17-a53f-4b89-be97-ea6fab7da2e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248909 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d069b2-6c32-49b8-8374-a6122355f4d3-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248942 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmv5d\" (UniqueName: \"kubernetes.io/projected/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-kube-api-access-qmv5d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248957 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzzr\" (UniqueName: \"kubernetes.io/projected/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-kube-api-access-mhzzr\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248971 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248982 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.248993 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebfa41f7-1d9b-4334-9b60-dafb65f462dd-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249002 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.249006 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5nd\" (UniqueName: \"kubernetes.io/projected/09d069b2-6c32-49b8-8374-a6122355f4d3-kube-api-access-cb5nd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.249364 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.249018 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fbfab17-a53f-4b89-be97-ea6fab7da2e7-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:18.829185 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.829154 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" Oct 30 08:58:18.829688 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.829158 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lztvw" event={"ID":"5fbfab17-a53f-4b89-be97-ea6fab7da2e7","Type":"ContainerDied","Data":"13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5"} Oct 30 08:58:18.829688 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.829287 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e0792fb82eb133bba20c42bb7fcacc6e744186129df9e7f9d81f96b56723b5" Oct 30 08:58:18.831352 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.831329 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" Oct 30 08:58:18.831479 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.831331 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pglk" event={"ID":"ebfa41f7-1d9b-4334-9b60-dafb65f462dd","Type":"ContainerDied","Data":"c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c"} Oct 30 08:58:18.831479 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.831430 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c75068396b1bbae2e3fc546cd2f0cf13ff669b5e22b8dffa507cbe48345e6c5c" Oct 30 08:58:18.833444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.833422 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" event={"ID":"a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb","Type":"ContainerDied","Data":"53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595"} Oct 30 08:58:18.833444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.833436 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ckrrt" Oct 30 08:58:18.833609 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.833465 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53159ea2d1854dbfe9dbb2b7cfb2465e7a7264bf993fc5e403be717379c63595" Oct 30 08:58:18.835550 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.835531 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" event={"ID":"09d069b2-6c32-49b8-8374-a6122355f4d3","Type":"ContainerDied","Data":"55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429"} Oct 30 08:58:18.835642 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.835569 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b77473d28b929993e00b94b0c38637a23c288a74a50533ece8a29a02a49429" Oct 30 08:58:18.836346 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.836153 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j5wjb" Oct 30 08:58:18.975741 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.975700 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:58:18.997315 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:18.997291 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:58:19.015191 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.015153 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:58:19.055751 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.055719 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\" (UniqueName: \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\") pod \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " Oct 30 08:58:19.055931 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.055782 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\") pod \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " Oct 30 08:58:19.055931 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.055818 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\" (UniqueName: \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\") pod \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " Oct 30 08:58:19.055931 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.055844 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\") pod \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " Oct 30 08:58:19.056189 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.056152 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\") pod \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\" (UID: \"021afb5a-edd8-4c17-9aaf-f4a1e45b9591\") " Oct 30 08:58:19.056339 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.056196 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\") pod \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\" (UID: \"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77\") " Oct 30 08:58:19.057052 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.057024 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle" pod "021afb5a-edd8-4c17-9aaf-f4a1e45b9591" (UID: "021afb5a-edd8-4c17-9aaf-f4a1e45b9591"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.057244 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.057203 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle" pod "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" (UID: "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.058300 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.058277 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj" pod "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" (UID: "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77"). InnerVolumeSpecName "kube-api-access-grgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:19.059184 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.059158 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl" pod "021afb5a-edd8-4c17-9aaf-f4a1e45b9591" (UID: "021afb5a-edd8-4c17-9aaf-f4a1e45b9591"). InnerVolumeSpecName "kube-api-access-bc8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:19.063332 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.063308 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util" pod "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" (UID: "b2754164-3d8d-4ab6-a9ea-5bc0af97ae77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.063780 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.063764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util" pod "021afb5a-edd8-4c17-9aaf-f4a1e45b9591" (UID: "021afb5a-edd8-4c17-9aaf-f4a1e45b9591"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.157195 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157123 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\") pod \"70557c79-8cc6-4877-9282-3f950e2a4f2c\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " Oct 30 08:58:19.157195 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157187 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\" (UniqueName: \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\") pod \"70557c79-8cc6-4877-9282-3f950e2a4f2c\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157232 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\") pod \"70557c79-8cc6-4877-9282-3f950e2a4f2c\" (UID: \"70557c79-8cc6-4877-9282-3f950e2a4f2c\") " Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157354 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157371 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgsj\" (UniqueName: \"kubernetes.io/projected/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-kube-api-access-grgsj\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157387 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157403 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157416 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2754164-3d8d-4ab6-a9ea-5bc0af97ae77-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157444 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157429 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8xl\" (UniqueName: \"kubernetes.io/projected/021afb5a-edd8-4c17-9aaf-f4a1e45b9591-kube-api-access-bc8xl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.157730 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.157625 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle" pod "70557c79-8cc6-4877-9282-3f950e2a4f2c" (UID: "70557c79-8cc6-4877-9282-3f950e2a4f2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.159286 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.159263 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs" pod "70557c79-8cc6-4877-9282-3f950e2a4f2c" (UID: "70557c79-8cc6-4877-9282-3f950e2a4f2c"). InnerVolumeSpecName "kube-api-access-z78rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:58:19.162716 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.162694 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util" pod "70557c79-8cc6-4877-9282-3f950e2a4f2c" (UID: "70557c79-8cc6-4877-9282-3f950e2a4f2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:58:19.258493 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.258452 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z78rs\" (UniqueName: \"kubernetes.io/projected/70557c79-8cc6-4877-9282-3f950e2a4f2c-kube-api-access-z78rs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.258493 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.258488 4920 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.258493 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.258498 4920 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70557c79-8cc6-4877-9282-3f950e2a4f2c-util\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 08:58:19.842014 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.841982 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" Oct 30 08:58:19.842014 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.841994 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qtwv8" event={"ID":"70557c79-8cc6-4877-9282-3f950e2a4f2c","Type":"ContainerDied","Data":"de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c"} Oct 30 08:58:19.842592 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.842041 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1668263f066ae6a13f47f89f85e5783f2d8c99339baf4509130c46798b606c" Oct 30 08:58:19.843934 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.843914 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" Oct 30 08:58:19.844058 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.843940 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7x227" event={"ID":"b2754164-3d8d-4ab6-a9ea-5bc0af97ae77","Type":"ContainerDied","Data":"a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8"} Oct 30 08:58:19.844058 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.843982 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c7dc4b3319dceb37f8504ea9f013f4e426b2a9a35d0868660a445afb68a9b8" Oct 30 08:58:19.845805 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.845784 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" event={"ID":"021afb5a-edd8-4c17-9aaf-f4a1e45b9591","Type":"ContainerDied","Data":"ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44"} Oct 30 08:58:19.845921 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.845822 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtjw6" Oct 30 08:58:19.845921 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:19.845822 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6f8440fd96528ed713fd52de3438903506959f8ce6e03df6969b846e5afd44" Oct 30 08:58:22.567338 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:22.567301 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:23.429011 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.428974 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g"] Oct 30 08:58:23.429302 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429288 4920 cpu_manager.go:414] "containerMap: removing container" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="util" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429304 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="util" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429312 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="util" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429317 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="util" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429326 4920 cpu_manager.go:414] "containerMap: removing container" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429332 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429337 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429343 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429353 4920 cpu_manager.go:414] "containerMap: removing container" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="pull" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429358 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="pull" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429363 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429368 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="extract" Oct 30 08:58:23.429371 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429376 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429380 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429387 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429392 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429399 4920 cpu_manager.go:414] "containerMap: removing container" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429403 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429409 4920 cpu_manager.go:414] "containerMap: removing container" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429413 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429420 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429424 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429430 4920 cpu_manager.go:414] "containerMap: removing container" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429434 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429439 4920 cpu_manager.go:414] "containerMap: removing container" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429443 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429452 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429457 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429464 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429469 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429474 4920 cpu_manager.go:414] "containerMap: removing container" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429478 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429485 4920 cpu_manager.go:414] "containerMap: removing container" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429489 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429495 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429499 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429509 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429513 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429519 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429523 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429529 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429534 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429544 4920 cpu_manager.go:414] "containerMap: removing container" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429549 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="util" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429555 4920 cpu_manager.go:414] "containerMap: removing container" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429559 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="pull" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429565 4920 cpu_manager.go:414] "containerMap: removing container" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429569 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429614 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="021afb5a-edd8-4c17-9aaf-f4a1e45b9591" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429622 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96030f5-5dc2-41cc-bf90-5fe5c6dd2abb" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429630 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d069b2-6c32-49b8-8374-a6122355f4d3" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429635 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="87367b4c-77d5-42d1-bcd6-29da099871f8" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429640 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfa41f7-1d9b-4334-9b60-dafb65f462dd" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429645 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbfab17-a53f-4b89-be97-ea6fab7da2e7" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429651 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2754164-3d8d-4ab6-a9ea-5bc0af97ae77" containerName="extract" Oct 30 08:58:23.429693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.429656 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="70557c79-8cc6-4877-9282-3f950e2a4f2c" containerName="extract" Oct 30 08:58:23.463828 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.463790 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.464158 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.464129 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g"] Oct 30 08:58:23.483579 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.483548 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 08:58:23.483703 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.483606 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-95xdf\"" Oct 30 08:58:23.483703 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.483691 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 08:58:23.483788 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.483721 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 08:58:23.483788 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.483742 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 08:58:23.583626 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.583586 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 08:58:23.593985 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.593954 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.594080 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.593991 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6nb\" (UniqueName: \"kubernetes.io/projected/c62f328d-1f05-4491-83a7-484a477b55a8-kube-api-access-fm6nb\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.594080 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.594019 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.594152 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.594100 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.594193 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.594153 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.594249 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.594209 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695167 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695081 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695167 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695141 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695170 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6nb\" (UniqueName: \"kubernetes.io/projected/c62f328d-1f05-4491-83a7-484a477b55a8-kube-api-access-fm6nb\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695202 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695258 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695283 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.695642 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.695611 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.697460 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.697426 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f328d-1f05-4491-83a7-484a477b55a8-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.697670 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.697653 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.697975 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.697951 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.698058 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.698032 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c62f328d-1f05-4491-83a7-484a477b55a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.712521 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.712495 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6nb\" (UniqueName: \"kubernetes.io/projected/c62f328d-1f05-4491-83a7-484a477b55a8-kube-api-access-fm6nb\") pod \"opendatahub-operator-controller-manager-69546f6877-9ml2g\" (UID: \"c62f328d-1f05-4491-83a7-484a477b55a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:23.808524 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:23.808489 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:24.033406 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:24.033326 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:24.180559 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:24.180522 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g"] Oct 30 08:58:24.184268 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:24.184242 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62f328d_1f05_4491_83a7_484a477b55a8.slice/crio-1be275dfa876fa8729a281e06faa14a9bd847edf32ea56ba8e75f2cff0a73706 WatchSource:0}: Error finding container 1be275dfa876fa8729a281e06faa14a9bd847edf32ea56ba8e75f2cff0a73706: Status 404 returned error can't find the container with id 1be275dfa876fa8729a281e06faa14a9bd847edf32ea56ba8e75f2cff0a73706 Oct 30 08:58:24.862410 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:24.862376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:24.870882 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:24.870847 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" event={"ID":"c62f328d-1f05-4491-83a7-484a477b55a8","Type":"ContainerStarted","Data":"1be275dfa876fa8729a281e06faa14a9bd847edf32ea56ba8e75f2cff0a73706"} Oct 30 08:58:26.354464 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:26.354427 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:26.455053 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:26.454987 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:27.325409 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:27.325379 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:27.889019 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:27.888985 4920 generic.go:356] "Generic (PLEG): container finished" podID="c62f328d-1f05-4491-83a7-484a477b55a8" containerID="7f7c92df6c51b0f690577638ea3923fcdad9d9c6de2c6f0cbbb8afe19a19487c" exitCode=0 Oct 30 08:58:27.889019 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:27.889020 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" event={"ID":"c62f328d-1f05-4491-83a7-484a477b55a8","Type":"ContainerDied","Data":"7f7c92df6c51b0f690577638ea3923fcdad9d9c6de2c6f0cbbb8afe19a19487c"} Oct 30 08:58:28.894764 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:28.894728 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" event={"ID":"c62f328d-1f05-4491-83a7-484a477b55a8","Type":"ContainerStarted","Data":"54b7f26f92d5d06523038f1b4219d7ad2bceb2f1aa4e998e61451c03112aac07"} Oct 30 08:58:28.895201 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:28.894879 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:28.981587 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:28.981534 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" podStartSLOduration=2.607131865 podStartE2EDuration="5.98151792s" totalImagesPullingTime="2.952854124s" totalInitContainerRuntime="421.531931ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:28.969737346 +0000 UTC m=+272.693164502" watchObservedRunningTime="2026-10-30 08:58:28.98151792 +0000 UTC m=+272.704945062" Oct 30 08:58:31.042974 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.042941 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl"] Oct 30 08:58:31.046341 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.046321 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.069811 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.069774 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:58:31.069811 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.069790 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hs2rs\"" Oct 30 08:58:31.069995 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.069872 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 08:58:31.072153 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.072121 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl"] Oct 30 08:58:31.133001 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:58:31.132963 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:58:31.160432 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.160401 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmbf\" (UniqueName: \"kubernetes.io/projected/6447bc92-6df2-45e7-a708-bce0609a4b6c-kube-api-access-6vmbf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.160574 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.160463 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6447bc92-6df2-45e7-a708-bce0609a4b6c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.261729 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.261698 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmbf\" (UniqueName: \"kubernetes.io/projected/6447bc92-6df2-45e7-a708-bce0609a4b6c-kube-api-access-6vmbf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.261882 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.261757 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6447bc92-6df2-45e7-a708-bce0609a4b6c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.262089 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.262073 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6447bc92-6df2-45e7-a708-bce0609a4b6c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.282947 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.282916 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmbf\" (UniqueName: \"kubernetes.io/projected/6447bc92-6df2-45e7-a708-bce0609a4b6c-kube-api-access-6vmbf\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl\" (UID: \"6447bc92-6df2-45e7-a708-bce0609a4b6c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.357256 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.357153 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" Oct 30 08:58:31.683162 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.683077 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl"] Oct 30 08:58:31.689858 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:31.689829 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6447bc92_6df2_45e7_a708_bce0609a4b6c.slice/crio-057522497aaa950ee7e2d884c1d555ca854d68e5b8cf31ff2b2b98fbae76e77d WatchSource:0}: Error finding container 057522497aaa950ee7e2d884c1d555ca854d68e5b8cf31ff2b2b98fbae76e77d: Status 404 returned error can't find the container with id 057522497aaa950ee7e2d884c1d555ca854d68e5b8cf31ff2b2b98fbae76e77d Oct 30 08:58:31.910347 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:31.910313 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" event={"ID":"6447bc92-6df2-45e7-a708-bce0609a4b6c","Type":"ContainerStarted","Data":"057522497aaa950ee7e2d884c1d555ca854d68e5b8cf31ff2b2b98fbae76e77d"} Oct 30 08:58:32.551871 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:32.551833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:32.654313 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:32.654277 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:33.183141 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:33.183112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:33.923260 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:33.923210 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" event={"ID":"6447bc92-6df2-45e7-a708-bce0609a4b6c","Type":"ContainerStarted","Data":"544a3fa1c8d227c3109ace1cf40f2e40e60be462145599a5f8d7db69821e1d44"} Oct 30 08:58:33.981369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:33.981306 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fmhcl" podStartSLOduration=1.548359163 podStartE2EDuration="2.981290911s" totalImagesPullingTime="1.432931748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:33.970143417 +0000 UTC m=+277.693570559" watchObservedRunningTime="2026-10-30 08:58:33.981290911 +0000 UTC m=+277.704718054" Oct 30 08:58:35.094424 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:35.094391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:35.206473 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:35.206421 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:36.261770 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.261734 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg"] Oct 30 08:58:36.265297 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.265280 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.278264 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.278244 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tlmdc\"" Oct 30 08:58:36.288514 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.288487 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 08:58:36.288627 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.288487 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 08:58:36.291340 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.291317 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg"] Oct 30 08:58:36.404438 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.404409 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.404615 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.404463 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md2q\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-kube-api-access-8md2q\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.505855 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.505814 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.506033 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.505876 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md2q\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-kube-api-access-8md2q\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.523513 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.523460 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.523620 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.523534 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md2q\" (UniqueName: \"kubernetes.io/projected/a03d8ac9-21cf-405b-a000-e9067021ee04-kube-api-access-8md2q\") pod \"cert-manager-webhook-5dbbbf75cc-tm7gg\" (UID: \"a03d8ac9-21cf-405b-a000-e9067021ee04\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.586798 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.586764 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:36.886939 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.886892 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg"] Oct 30 08:58:36.891037 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:36.891007 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda03d8ac9_21cf_405b_a000_e9067021ee04.slice/crio-2a81df4e5b0c809a1bfd807e4976bf4468a5d6b86556715bf79724a235038c36 WatchSource:0}: Error finding container 2a81df4e5b0c809a1bfd807e4976bf4468a5d6b86556715bf79724a235038c36: Status 404 returned error can't find the container with id 2a81df4e5b0c809a1bfd807e4976bf4468a5d6b86556715bf79724a235038c36 Oct 30 08:58:36.937543 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:36.937508 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" event={"ID":"a03d8ac9-21cf-405b-a000-e9067021ee04","Type":"ContainerStarted","Data":"2a81df4e5b0c809a1bfd807e4976bf4468a5d6b86556715bf79724a235038c36"} Oct 30 08:58:37.375155 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:37.375118 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:37.479516 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:37.479475 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:37.575399 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:37.575358 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:37.715828 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:37.715749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:38.373312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.373277 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg"] Oct 30 08:58:38.376733 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.376707 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:38.399932 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.399906 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 08:58:38.400089 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.400002 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 08:58:38.400168 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.400111 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tg86k\"" Oct 30 08:58:38.402567 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.402546 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg"] Oct 30 08:58:38.524892 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.524852 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzhp\" (UniqueName: \"kubernetes.io/projected/753f3050-bd21-4575-997c-157765af79b3-kube-api-access-5dzhp\") pod \"limitador-operator-controller-manager-5778bb4449-4wqhg\" (UID: \"753f3050-bd21-4575-997c-157765af79b3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:38.626304 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.626165 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzhp\" (UniqueName: \"kubernetes.io/projected/753f3050-bd21-4575-997c-157765af79b3-kube-api-access-5dzhp\") pod \"limitador-operator-controller-manager-5778bb4449-4wqhg\" (UID: \"753f3050-bd21-4575-997c-157765af79b3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:38.651609 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.651576 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzhp\" (UniqueName: \"kubernetes.io/projected/753f3050-bd21-4575-997c-157765af79b3-kube-api-access-5dzhp\") pod \"limitador-operator-controller-manager-5778bb4449-4wqhg\" (UID: \"753f3050-bd21-4575-997c-157765af79b3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:38.691364 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.691329 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:38.998281 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:38.998185 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg"] Oct 30 08:58:39.001943 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:39.001913 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753f3050_bd21_4575_997c_157765af79b3.slice/crio-44318b5f67933877afefc183fbbc6d8f3046e2eb386a376e2a58a1e4772bd5b2 WatchSource:0}: Error finding container 44318b5f67933877afefc183fbbc6d8f3046e2eb386a376e2a58a1e4772bd5b2: Status 404 returned error can't find the container with id 44318b5f67933877afefc183fbbc6d8f3046e2eb386a376e2a58a1e4772bd5b2 Oct 30 08:58:39.517897 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:39.517861 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:39.903026 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:39.902989 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-9ml2g" Oct 30 08:58:39.957004 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:39.956967 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" event={"ID":"753f3050-bd21-4575-997c-157765af79b3","Type":"ContainerStarted","Data":"44318b5f67933877afefc183fbbc6d8f3046e2eb386a376e2a58a1e4772bd5b2"} Oct 30 08:58:39.959272 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:39.959241 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" event={"ID":"a03d8ac9-21cf-405b-a000-e9067021ee04","Type":"ContainerStarted","Data":"33878a7c43e3adc5144e73e5d495e0ccfd428ad2b5e092e82208362a6b1636e2"} Oct 30 08:58:39.959437 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:39.959422 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:40.019264 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:40.019184 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" podStartSLOduration=1.5728415390000001 podStartE2EDuration="4.019162547s" totalImagesPullingTime="2.446321008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:40.017196574 +0000 UTC m=+283.740623718" watchObservedRunningTime="2026-10-30 08:58:40.019162547 +0000 UTC m=+283.742589691" Oct 30 08:58:41.524553 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.524459 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx"] Oct 30 08:58:41.528009 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.527992 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:41.550814 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.550780 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2g4\" (UniqueName: \"kubernetes.io/projected/20210cd6-62e9-436c-bc29-84bc56bb2589-kube-api-access-gs2g4\") pod \"dns-operator-controller-manager-8557cbdb84-slvzx\" (UID: \"20210cd6-62e9-436c-bc29-84bc56bb2589\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:41.559374 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.559341 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vbnqf\"" Oct 30 08:58:41.559528 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.559425 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 08:58:41.561801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.561777 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx"] Oct 30 08:58:41.652161 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.652122 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2g4\" (UniqueName: \"kubernetes.io/projected/20210cd6-62e9-436c-bc29-84bc56bb2589-kube-api-access-gs2g4\") pod \"dns-operator-controller-manager-8557cbdb84-slvzx\" (UID: \"20210cd6-62e9-436c-bc29-84bc56bb2589\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:41.674449 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.674416 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2g4\" (UniqueName: \"kubernetes.io/projected/20210cd6-62e9-436c-bc29-84bc56bb2589-kube-api-access-gs2g4\") pod \"dns-operator-controller-manager-8557cbdb84-slvzx\" (UID: \"20210cd6-62e9-436c-bc29-84bc56bb2589\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:41.840627 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.840587 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:41.971528 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.971493 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" event={"ID":"753f3050-bd21-4575-997c-157765af79b3","Type":"ContainerStarted","Data":"35b5aac8496a12c5d01f379c19e0a6bce7b973b8ecb367fd70467d0406fb7176"} Oct 30 08:58:41.971733 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:41.971566 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:42.009749 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:42.009699 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" podStartSLOduration=1.782003937 podStartE2EDuration="4.009685856s" totalImagesPullingTime="2.227681919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:42.007606288 +0000 UTC m=+285.731033431" watchObservedRunningTime="2026-10-30 08:58:42.009685856 +0000 UTC m=+285.733112999" Oct 30 08:58:42.113600 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:42.113517 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:42.141026 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:42.138487 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx"] Oct 30 08:58:42.147783 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:42.147753 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20210cd6_62e9_436c_bc29_84bc56bb2589.slice/crio-a8f2108882fe891c10000a5f919c5679a1c243cdd0a207901a49a8ab3f1e1635 WatchSource:0}: Error finding container a8f2108882fe891c10000a5f919c5679a1c243cdd0a207901a49a8ab3f1e1635: Status 404 returned error can't find the container with id a8f2108882fe891c10000a5f919c5679a1c243cdd0a207901a49a8ab3f1e1635 Oct 30 08:58:42.937537 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:42.937493 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:42.980097 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:42.980034 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" event={"ID":"20210cd6-62e9-436c-bc29-84bc56bb2589","Type":"ContainerStarted","Data":"a8f2108882fe891c10000a5f919c5679a1c243cdd0a207901a49a8ab3f1e1635"} Oct 30 08:58:43.073458 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:43.073424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:43.985157 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:43.985115 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" event={"ID":"20210cd6-62e9-436c-bc29-84bc56bb2589","Type":"ContainerStarted","Data":"b7e589e13e046cf48fa4fac50ca1fbe76db07382a93179895b960329f7d13b37"} Oct 30 08:58:43.985574 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:43.985176 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:44.024010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:44.023958 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" podStartSLOduration=1.739490848 podStartE2EDuration="3.023941077s" totalImagesPullingTime="1.284450229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:44.022876437 +0000 UTC m=+287.746303581" watchObservedRunningTime="2026-10-30 08:58:44.023941077 +0000 UTC m=+287.747368221" Oct 30 08:58:44.738843 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:44.738808 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:44.840256 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:44.840209 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:45.353406 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:45.353367 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:45.968040 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:45.968007 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tm7gg" Oct 30 08:58:46.140635 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.140593 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:46.388568 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.388527 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dscb"] Oct 30 08:58:46.393409 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.393385 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:46.409851 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.409822 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-68j22\"" Oct 30 08:58:46.447110 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.447079 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dscb"] Oct 30 08:58:46.492407 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.492370 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kbh\" (UniqueName: \"kubernetes.io/projected/97ce6c5e-2a90-43bb-8c99-57a28daf5338-kube-api-access-d9kbh\") pod \"authorino-operator-7b7bb4bc77-8dscb\" (UID: \"97ce6c5e-2a90-43bb-8c99-57a28daf5338\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:46.593801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.593762 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kbh\" (UniqueName: \"kubernetes.io/projected/97ce6c5e-2a90-43bb-8c99-57a28daf5338-kube-api-access-d9kbh\") pod \"authorino-operator-7b7bb4bc77-8dscb\" (UID: \"97ce6c5e-2a90-43bb-8c99-57a28daf5338\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:46.610856 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.610823 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kbh\" (UniqueName: \"kubernetes.io/projected/97ce6c5e-2a90-43bb-8c99-57a28daf5338-kube-api-access-d9kbh\") pod \"authorino-operator-7b7bb4bc77-8dscb\" (UID: \"97ce6c5e-2a90-43bb-8c99-57a28daf5338\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:46.706306 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.706224 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:46.743652 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.743607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:46.909177 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:46.909142 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:47.009076 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:47.008986 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dscb"] Oct 30 08:58:47.013101 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:47.013060 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ce6c5e_2a90_43bb_8c99_57a28daf5338.slice/crio-649ea806c2a0609af52e9aebdb91363d6fa87c2b368eaae8de0530b397f50739 WatchSource:0}: Error finding container 649ea806c2a0609af52e9aebdb91363d6fa87c2b368eaae8de0530b397f50739: Status 404 returned error can't find the container with id 649ea806c2a0609af52e9aebdb91363d6fa87c2b368eaae8de0530b397f50739 Oct 30 08:58:47.786895 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:47.786848 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:48.006517 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:48.006484 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" event={"ID":"97ce6c5e-2a90-43bb-8c99-57a28daf5338","Type":"ContainerStarted","Data":"649ea806c2a0609af52e9aebdb91363d6fa87c2b368eaae8de0530b397f50739"} Oct 30 08:58:48.519584 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:48.519550 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:48.983386 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:48.983348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:49.014717 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:49.014682 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" event={"ID":"97ce6c5e-2a90-43bb-8c99-57a28daf5338","Type":"ContainerStarted","Data":"2b50852a9b3c1d0c82c23d7414dbf81877b91b967b6d04fea784919cc5ba53fd"} Oct 30 08:58:49.014912 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:49.014802 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:58:49.049789 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:49.049737 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" podStartSLOduration=1.7610680589999999 podStartE2EDuration="3.049724098s" totalImagesPullingTime="1.288656039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:49.049080552 +0000 UTC m=+292.772507705" watchObservedRunningTime="2026-10-30 08:58:49.049724098 +0000 UTC m=+292.773151241" Oct 30 08:58:50.366849 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:50.366813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:50.528009 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:50.527976 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:52.276902 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.276866 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:52.359536 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.359498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:52.770405 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.770363 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 08:58:52.774162 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.774138 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.787301 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.787276 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 08:58:52.795396 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.795371 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xr9hl\"" Oct 30 08:58:52.842549 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.842468 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.842549 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.842502 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lcv\" (UniqueName: \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.943316 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.943276 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.943316 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.943312 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lcv\" (UniqueName: \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.943795 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.943769 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.960499 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.960466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lcv\" (UniqueName: \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\") pod \"kuadrant-operator-controller-manager-678d87766f-v4v2x\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:52.982654 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:52.982629 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4wqhg" Oct 30 08:58:53.086634 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:53.086592 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:53.400412 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:53.400334 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 08:58:53.404620 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:53.404593 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01119c86_dc03_4660_9fff_2f027184b4ff.slice/crio-c3fe8ab95b759a96831768a89a55453c86462e298afc11bbb46bf489735e4741 WatchSource:0}: Error finding container c3fe8ab95b759a96831768a89a55453c86462e298afc11bbb46bf489735e4741: Status 404 returned error can't find the container with id c3fe8ab95b759a96831768a89a55453c86462e298afc11bbb46bf489735e4741 Oct 30 08:58:54.041348 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:54.041300 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" event={"ID":"01119c86-dc03-4660-9fff-2f027184b4ff","Type":"ContainerStarted","Data":"c3fe8ab95b759a96831768a89a55453c86462e298afc11bbb46bf489735e4741"} Oct 30 08:58:54.919096 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:54.919052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:54.993173 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:54.993135 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-slvzx" Oct 30 08:58:55.021996 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:55.021960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:56.428929 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:56.428895 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 08:58:56.429441 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:56.429158 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 08:58:57.318106 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:57.318064 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:57.425360 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:57.425327 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:57.625097 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:57.625014 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:58.064819 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.064786 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" event={"ID":"01119c86-dc03-4660-9fff-2f027184b4ff","Type":"ContainerStarted","Data":"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22"} Oct 30 08:58:58.065015 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.064909 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:58:58.089661 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.089627 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:58.134974 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.134926 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" podStartSLOduration=2.07320177 podStartE2EDuration="6.134913805s" totalImagesPullingTime="4.061712035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:58.12309267 +0000 UTC m=+301.846519814" watchObservedRunningTime="2026-10-30 08:58:58.134913805 +0000 UTC m=+301.858340948" Oct 30 08:58:58.199051 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.198983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:58.311411 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.311375 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2"] Oct 30 08:58:58.315083 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.315020 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.330012 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.329983 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gd5ml\"" Oct 30 08:58:58.336407 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.336375 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2"] Oct 30 08:58:58.387053 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.387021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:58.394966 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.394943 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395108 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.394977 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395108 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395004 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395186 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395108 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npkhc\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-kube-api-access-npkhc\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395186 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395145 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395186 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395170 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395315 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395281 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8218b15b-003a-4f9c-bed8-c74889c32f46-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395409 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395389 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.395452 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.395436 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496067 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496029 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496080 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npkhc\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-kube-api-access-npkhc\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496105 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496130 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496147 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8218b15b-003a-4f9c-bed8-c74889c32f46-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496241 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496275 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496312 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496304 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496731 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496334 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496731 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496542 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496731 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496708 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496952 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496918 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.496952 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.496945 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.497286 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.497259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:58.497336 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.497296 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8218b15b-003a-4f9c-bed8-c74889c32f46-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.498730 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.498704 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.498834 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.498815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.513453 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.513421 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.513564 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.513455 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npkhc\" (UniqueName: \"kubernetes.io/projected/8218b15b-003a-4f9c-bed8-c74889c32f46-kube-api-access-npkhc\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-lndw2\" (UID: \"8218b15b-003a-4f9c-bed8-c74889c32f46\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.628730 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.628639 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:58.950571 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.950481 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2"] Oct 30 08:58:58.956767 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:58:58.956741 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8218b15b_003a_4f9c_bed8_c74889c32f46.slice/crio-3d90f0693b991d439172dc666b23a7ca9812d49e7782b391b6ab475d359ed9bb WatchSource:0}: Error finding container 3d90f0693b991d439172dc666b23a7ca9812d49e7782b391b6ab475d359ed9bb: Status 404 returned error can't find the container with id 3d90f0693b991d439172dc666b23a7ca9812d49e7782b391b6ab475d359ed9bb Oct 30 08:58:58.959059 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.959029 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:58:58.959134 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.959098 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:58:58.959134 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:58.959124 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:58:59.072951 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.072913 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" event={"ID":"8218b15b-003a-4f9c-bed8-c74889c32f46","Type":"ContainerStarted","Data":"9dc176c0b009a286fb214b9b487dc8ceffd4024a5930c23bad123c12a4798941"} Oct 30 08:58:59.073109 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.072976 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" event={"ID":"8218b15b-003a-4f9c-bed8-c74889c32f46","Type":"ContainerStarted","Data":"3d90f0693b991d439172dc666b23a7ca9812d49e7782b391b6ab475d359ed9bb"} Oct 30 08:58:59.112708 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.112657 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" podStartSLOduration=1.112641444 podStartE2EDuration="1.112641444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:59.11159104 +0000 UTC m=+302.835018225" watchObservedRunningTime="2026-10-30 08:58:59.112641444 +0000 UTC m=+302.836068583" Oct 30 08:58:59.354372 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.354333 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:59.454092 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.454050 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:58:59.629584 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.629508 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:58:59.743006 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:58:59.742970 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:00.021128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:00.021043 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dscb" Oct 30 08:59:00.634109 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:00.634077 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:59:00.895103 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:00.895024 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:01.083332 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:01.083301 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:59:01.084276 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:01.084256 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-lndw2" Oct 30 08:59:01.136371 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:59:01.136329 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:59:01.996641 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:01.996605 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c"] Oct 30 08:59:01.999911 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:01.999889 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.012443 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.012421 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 08:59:02.012557 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.012469 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 08:59:02.022402 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.022368 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-blpwc\"" Oct 30 08:59:02.043621 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.043576 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 08:59:02.045808 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.045785 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c"] Oct 30 08:59:02.129942 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.129900 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c7a1f0e-704f-4653-9cfc-2a1aef054240-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.130120 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.130048 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.130185 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.130165 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cm9l\" (UniqueName: \"kubernetes.io/projected/1c7a1f0e-704f-4653-9cfc-2a1aef054240-kube-api-access-4cm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.130290 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.130270 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.231515 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.231474 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.231693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.231542 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c7a1f0e-704f-4653-9cfc-2a1aef054240-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.231693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.231580 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.231693 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.231599 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cm9l\" (UniqueName: \"kubernetes.io/projected/1c7a1f0e-704f-4653-9cfc-2a1aef054240-kube-api-access-4cm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.232059 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.232031 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.232059 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.232048 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7a1f0e-704f-4653-9cfc-2a1aef054240-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.232628 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.232602 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c7a1f0e-704f-4653-9cfc-2a1aef054240-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.248756 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.248686 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cm9l\" (UniqueName: \"kubernetes.io/projected/1c7a1f0e-704f-4653-9cfc-2a1aef054240-kube-api-access-4cm9l\") pod \"kserve-module-controller-manager-7566cf5fc8-b6z6c\" (UID: \"1c7a1f0e-704f-4653-9cfc-2a1aef054240\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.312729 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.312690 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:02.626018 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.625989 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c"] Oct 30 08:59:02.629437 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:59:02.629407 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7a1f0e_704f_4653_9cfc_2a1aef054240.slice/crio-cdf8f1e5d35b2b0a3ff6d326f2c3fd95ad89367269b9c614fc69701e0b41ebb8 WatchSource:0}: Error finding container cdf8f1e5d35b2b0a3ff6d326f2c3fd95ad89367269b9c614fc69701e0b41ebb8: Status 404 returned error can't find the container with id cdf8f1e5d35b2b0a3ff6d326f2c3fd95ad89367269b9c614fc69701e0b41ebb8 Oct 30 08:59:02.631359 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:02.631339 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:59:03.095413 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:03.095367 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" event={"ID":"1c7a1f0e-704f-4653-9cfc-2a1aef054240","Type":"ContainerStarted","Data":"cdf8f1e5d35b2b0a3ff6d326f2c3fd95ad89367269b9c614fc69701e0b41ebb8"} Oct 30 08:59:04.333657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:04.333614 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:04.418805 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:04.418769 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:04.520864 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:04.520828 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:04.609585 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:04.609494 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:05.111476 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:05.111437 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" event={"ID":"1c7a1f0e-704f-4653-9cfc-2a1aef054240","Type":"ContainerStarted","Data":"c19619a98b84c48eda23bcb3b44d5bc13f17a80646bd6f6a5b44cb3e42c93f53"} Oct 30 08:59:05.111657 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:05.111522 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:05.161457 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:05.161404 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" podStartSLOduration=2.071414776 podStartE2EDuration="4.161388845s" totalImagesPullingTime="2.089974069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:05.159434279 +0000 UTC m=+308.882861435" watchObservedRunningTime="2026-10-30 08:59:05.161388845 +0000 UTC m=+308.884816019" Oct 30 08:59:06.225424 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:06.225392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:06.331352 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:06.331301 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:09.075825 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:09.075787 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 08:59:09.619671 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:09.619635 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:09.754949 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:09.754912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:10.142369 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:10.142337 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:10.248010 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:10.247971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:10.858373 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:10.858336 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:11.026707 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:11.026673 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:18.522810 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:18.522769 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:18.682352 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:18.682317 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:21.528098 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:21.528066 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:21.633712 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:21.633676 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:26.118666 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:26.118630 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b6z6c" Oct 30 08:59:26.407060 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:26.406978 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:26.532762 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:26.532721 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:26.644900 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:26.644865 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:31.134257 ip-10-0-129-105 kubenswrapper[4920]: E1030 08:59:31.134206 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 08:59:31.490779 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:31.490695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:31.592993 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:31.592958 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:46.269838 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:46.269783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:56.692277 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.692238 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:56.792631 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.792588 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:56.894716 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.894676 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:56.962102 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.962026 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28h4p"] Oct 30 08:59:56.969622 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.969594 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:56.984594 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.984567 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28h4p"] Oct 30 08:59:56.987602 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:56.987573 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:57.024235 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.024194 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v5t5r\"" Oct 30 08:59:57.024235 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.024199 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 08:59:57.027706 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.027671 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbq2f\" (UniqueName: \"kubernetes.io/projected/eaf9b82c-6b5f-454c-a452-747042200b5b-kube-api-access-gbq2f\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.027828 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.027814 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b82c-6b5f-454c-a452-747042200b5b-cert\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.042867 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.042837 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mpftf"] Oct 30 08:59:57.048228 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.048194 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.066791 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.066763 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mpftf"] Oct 30 08:59:57.072711 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.072688 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 08:59:57.072844 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.072806 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dz5bd\"" Oct 30 08:59:57.128491 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.128458 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbq2f\" (UniqueName: \"kubernetes.io/projected/eaf9b82c-6b5f-454c-a452-747042200b5b-kube-api-access-gbq2f\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.128686 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.128538 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d00f523-413e-4a63-800f-1a813a802d36-tls-certs\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.128686 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.128588 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b82c-6b5f-454c-a452-747042200b5b-cert\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.128814 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.128678 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fz6\" (UniqueName: \"kubernetes.io/projected/9d00f523-413e-4a63-800f-1a813a802d36-kube-api-access-z4fz6\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.130988 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.130963 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b82c-6b5f-454c-a452-747042200b5b-cert\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.148926 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.148893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbq2f\" (UniqueName: \"kubernetes.io/projected/eaf9b82c-6b5f-454c-a452-747042200b5b-kube-api-access-gbq2f\") pod \"kserve-controller-manager-776cc66d47-28h4p\" (UID: \"eaf9b82c-6b5f-454c-a452-747042200b5b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.153032 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.153008 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:57.229378 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.229297 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d00f523-413e-4a63-800f-1a813a802d36-tls-certs\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.229378 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.229368 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fz6\" (UniqueName: \"kubernetes.io/projected/9d00f523-413e-4a63-800f-1a813a802d36-kube-api-access-z4fz6\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.231801 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.231770 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d00f523-413e-4a63-800f-1a813a802d36-tls-certs\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.247008 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.246981 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fz6\" (UniqueName: \"kubernetes.io/projected/9d00f523-413e-4a63-800f-1a813a802d36-kube-api-access-z4fz6\") pod \"model-serving-api-75bd89fdfc-mpftf\" (UID: \"9d00f523-413e-4a63-800f-1a813a802d36\") " pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.259887 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.259860 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:57.284761 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.284730 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 08:59:57.361753 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.361721 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:57.456631 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.456592 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:57.597275 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.596242 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28h4p"] Oct 30 08:59:57.603072 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:59:57.603045 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf9b82c_6b5f_454c_a452_747042200b5b.slice/crio-c0bf4e731561ea2dba66c4f937200fdfa4ab0f5f84857155a8372bffe005ad79 WatchSource:0}: Error finding container c0bf4e731561ea2dba66c4f937200fdfa4ab0f5f84857155a8372bffe005ad79: Status 404 returned error can't find the container with id c0bf4e731561ea2dba66c4f937200fdfa4ab0f5f84857155a8372bffe005ad79 Oct 30 08:59:57.676033 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:57.675999 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mpftf"] Oct 30 08:59:57.679765 ip-10-0-129-105 kubenswrapper[4920]: W1030 08:59:57.679735 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d00f523_413e_4a63_800f_1a813a802d36.slice/crio-a230e26a1acbe4848f9b3ad7543b3f7c7192886abee859894ed772536429f24e WatchSource:0}: Error finding container a230e26a1acbe4848f9b3ad7543b3f7c7192886abee859894ed772536429f24e: Status 404 returned error can't find the container with id a230e26a1acbe4848f9b3ad7543b3f7c7192886abee859894ed772536429f24e Oct 30 08:59:58.383346 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:58.383202 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" event={"ID":"9d00f523-413e-4a63-800f-1a813a802d36","Type":"ContainerStarted","Data":"a230e26a1acbe4848f9b3ad7543b3f7c7192886abee859894ed772536429f24e"} Oct 30 08:59:58.385566 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:58.385473 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" event={"ID":"eaf9b82c-6b5f-454c-a452-747042200b5b","Type":"ContainerStarted","Data":"c0bf4e731561ea2dba66c4f937200fdfa4ab0f5f84857155a8372bffe005ad79"} Oct 30 08:59:59.203325 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.203111 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:59.294979 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.294945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:59.376377 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.376340 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:59.393128 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.393087 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" event={"ID":"9d00f523-413e-4a63-800f-1a813a802d36","Type":"ContainerStarted","Data":"2a23aa480799997161bc964dbedfa0f211e5006212a2a936b19159aaef88628f"} Oct 30 08:59:59.393570 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.393252 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 08:59:59.456473 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.456363 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" podStartSLOduration=0.897666312 podStartE2EDuration="2.456347276s" totalImagesPullingTime="1.558680964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:59.455259983 +0000 UTC m=+363.178687140" watchObservedRunningTime="2026-10-30 08:59:59.456347276 +0000 UTC m=+363.179774419" Oct 30 08:59:59.509783 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.509753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 08:59:59.610410 ip-10-0-129-105 kubenswrapper[4920]: I1030 08:59:59.610385 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:00:00.400530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:00.400493 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" event={"ID":"eaf9b82c-6b5f-454c-a452-747042200b5b","Type":"ContainerStarted","Data":"abec288b7e9b86aa3f45a75c4a02b1e133df9841a9915998bc19c434cd8bc85c"} Oct 30 09:00:00.434968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:00.434921 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" podStartSLOduration=2.512908461 podStartE2EDuration="4.434907052s" totalImagesPullingTime="1.921998591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:00:00.433085339 +0000 UTC m=+364.156512500" watchObservedRunningTime="2026-10-30 09:00:00.434907052 +0000 UTC m=+364.158334209" Oct 30 09:00:01.134463 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:00:01.134424 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:00:01.406340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:01.406255 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 09:00:10.405745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:10.405714 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mpftf" Oct 30 09:00:31.133735 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:00:31.133693 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:00:32.415500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:32.415464 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-28h4p" Oct 30 09:00:46.369424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:00:46.369335 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:01:01.135545 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:01:01.135507 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:01:31.133939 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:01:31.133896 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:01:46.481353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:01:46.481319 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:01:54.013832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:01:54.013795 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:01:54.093426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:01:54.093396 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:01:56.517595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:01:56.517559 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:01:56.610558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:01:56.610510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:02:01.134350 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:02:01.134307 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:02:31.134052 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:02:31.134015 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:02:46.612868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:02:46.612816 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:03:01.133911 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:03:01.133873 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:03:24.981091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:03:24.981057 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:03:31.133739 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:03:31.133691 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:03:46.699583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:03:46.699491 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:03:56.466310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:03:56.466281 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:03:56.469341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:03:56.469317 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:04:01.132954 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:04:01.132919 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:04:07.722791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:04:07.722754 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:04:25.754499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:04:25.754461 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:04:31.134628 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:04:31.134580 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:04:46.826964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:04:46.826916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:04:53.149838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:04:53.149805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:05:01.133990 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:05:01.133953 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:05:31.134540 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:05:31.134501 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:05:36.521342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:05:36.521303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:05:36.621636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:05:36.621598 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:05:45.285366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:05:45.285327 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:05:45.365399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:05:45.365368 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:05:46.946386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:05:46.946350 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:06:01.133658 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:06:01.133620 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:06:31.132857 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:06:31.132811 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:06:47.062709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:06:47.062628 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:07:01.134198 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:07:01.134154 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:07:31.133491 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:07:31.133453 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:07:47.183416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:07:47.183382 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:08:01.134301 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:08:01.134264 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:08:31.133394 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:08:31.133356 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:08:34.573956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:08:34.573911 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:08:47.278404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:08:47.278371 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:08:56.509048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:08:56.509014 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:08:56.510791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:08:56.510768 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:09:01.134110 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:09:01.134072 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:09:18.260772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.260734 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v4v2x_01119c86-dc03-4660-9fff-2f027184b4ff/manager/0.log" Oct 30 09:09:18.499764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.499726 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 09:09:18.499960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.499936 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" podUID="01119c86-dc03-4660-9fff-2f027184b4ff" containerName="manager" containerID="cri-o://8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22" gracePeriod=10 Oct 30 09:09:18.512522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.512493 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2"] Oct 30 09:09:18.520328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.520308 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.547279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.547248 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2"] Oct 30 09:09:18.605339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.605304 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04274227-f7ec-4b45-b118-958c96a7b20c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.605511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.605392 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwv4\" (UniqueName: \"kubernetes.io/projected/04274227-f7ec-4b45-b118-958c96a7b20c-kube-api-access-bjwv4\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.706841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.706806 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04274227-f7ec-4b45-b118-958c96a7b20c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.707042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.706851 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwv4\" (UniqueName: \"kubernetes.io/projected/04274227-f7ec-4b45-b118-958c96a7b20c-kube-api-access-bjwv4\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.707276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.707249 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04274227-f7ec-4b45-b118-958c96a7b20c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.727800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.727766 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwv4\" (UniqueName: \"kubernetes.io/projected/04274227-f7ec-4b45-b118-958c96a7b20c-kube-api-access-bjwv4\") pod \"kuadrant-operator-controller-manager-678d87766f-n98s2\" (UID: \"04274227-f7ec-4b45-b118-958c96a7b20c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.749333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.749310 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 09:09:18.807917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.807884 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\" (UniqueName: \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\") pod \"01119c86-dc03-4660-9fff-2f027184b4ff\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " Oct 30 09:09:18.808117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.807944 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\") pod \"01119c86-dc03-4660-9fff-2f027184b4ff\" (UID: \"01119c86-dc03-4660-9fff-2f027184b4ff\") " Oct 30 09:09:18.808468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.808434 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume" pod "01119c86-dc03-4660-9fff-2f027184b4ff" (UID: "01119c86-dc03-4660-9fff-2f027184b4ff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:09:18.810109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.810084 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv" pod "01119c86-dc03-4660-9fff-2f027184b4ff" (UID: "01119c86-dc03-4660-9fff-2f027184b4ff"). InnerVolumeSpecName "kube-api-access-l9lcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:09:18.895875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.895835 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:18.909203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.909174 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lcv\" (UniqueName: \"kubernetes.io/projected/01119c86-dc03-4660-9fff-2f027184b4ff-kube-api-access-l9lcv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:09:18.909203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:18.909203 4920 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01119c86-dc03-4660-9fff-2f027184b4ff-extensions-socket-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:09:19.207152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.207120 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2"] Oct 30 09:09:19.209127 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:09:19.209096 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04274227_f7ec_4b45_b118_958c96a7b20c.slice/crio-26a18042ea6d9c7fb83d6089c912a018a7cda2c1fcf09ee38d63f737c9196e76 WatchSource:0}: Error finding container 26a18042ea6d9c7fb83d6089c912a018a7cda2c1fcf09ee38d63f737c9196e76: Status 404 returned error can't find the container with id 26a18042ea6d9c7fb83d6089c912a018a7cda2c1fcf09ee38d63f737c9196e76 Oct 30 09:09:19.224090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.224060 4920 generic.go:356] "Generic (PLEG): container finished" podID="01119c86-dc03-4660-9fff-2f027184b4ff" containerID="8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22" exitCode=0 Oct 30 09:09:19.224203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.224110 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" Oct 30 09:09:19.224203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.224154 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" event={"ID":"01119c86-dc03-4660-9fff-2f027184b4ff","Type":"ContainerDied","Data":"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22"} Oct 30 09:09:19.224349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.224207 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x" event={"ID":"01119c86-dc03-4660-9fff-2f027184b4ff","Type":"ContainerDied","Data":"c3fe8ab95b759a96831768a89a55453c86462e298afc11bbb46bf489735e4741"} Oct 30 09:09:19.224349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.224248 4920 scope.go:122] "RemoveContainer" containerID="8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22" Oct 30 09:09:19.225851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.225824 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" event={"ID":"04274227-f7ec-4b45-b118-958c96a7b20c","Type":"ContainerStarted","Data":"26a18042ea6d9c7fb83d6089c912a018a7cda2c1fcf09ee38d63f737c9196e76"} Oct 30 09:09:19.234033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.234014 4920 scope.go:122] "RemoveContainer" containerID="8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22" Oct 30 09:09:19.234416 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:09:19.234395 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22\": container with ID starting with 8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22 not found: ID does not exist" containerID="8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22" Oct 30 09:09:19.234481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.234424 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22"} err="failed to get container status \"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22\": rpc error: code = NotFound desc = could not find container \"8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22\": container with ID starting with 8fd8e9b9ef2ae9105dea011f56a9606566be3242a9f94d417737e07223242f22 not found: ID does not exist" Oct 30 09:09:19.277362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.277330 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 09:09:19.280034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:19.280010 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v4v2x"] Oct 30 09:09:20.238024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:20.237969 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" event={"ID":"04274227-f7ec-4b45-b118-958c96a7b20c","Type":"ContainerStarted","Data":"b07a0edb3c8923382724fdd530e67085fc4799f85757e387babd0fe84e69cd11"} Oct 30 09:09:20.238211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:20.238063 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:20.298861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:20.298390 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" podStartSLOduration=2.298371162 podStartE2EDuration="2.298371162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:20.286593199 +0000 UTC m=+924.010020342" watchObservedRunningTime="2026-10-30 09:09:20.298371162 +0000 UTC m=+924.021798306" Oct 30 09:09:20.396639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:20.396605 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01119c86-dc03-4660-9fff-2f027184b4ff" path="/var/lib/kubelet/pods/01119c86-dc03-4660-9fff-2f027184b4ff/volumes" Oct 30 09:09:31.133974 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:09:31.133936 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:09:31.247443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:31.247413 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n98s2" Oct 30 09:09:37.339400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.339362 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:37.405430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.405397 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf"] Oct 30 09:09:37.405790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.405777 4920 cpu_manager.go:414] "containerMap: removing container" podUID="01119c86-dc03-4660-9fff-2f027184b4ff" containerName="manager" Oct 30 09:09:37.405858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.405792 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01119c86-dc03-4660-9fff-2f027184b4ff" containerName="manager" Oct 30 09:09:37.405917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.405861 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="01119c86-dc03-4660-9fff-2f027184b4ff" containerName="manager" Oct 30 09:09:37.408430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.408406 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.426190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.426154 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf"] Oct 30 09:09:37.441242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.441178 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:37.462651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.462612 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 09:09:37.463344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.463308 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 09:09:37.463491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.463398 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2zmf2\"" Oct 30 09:09:37.469265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.469210 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca4979b8-86d1-4bc9-b06b-ad36059c1279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.469423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.469302 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca4979b8-86d1-4bc9-b06b-ad36059c1279-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.469423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.469357 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxgv\" (UniqueName: \"kubernetes.io/projected/ca4979b8-86d1-4bc9-b06b-ad36059c1279-kube-api-access-7jxgv\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.536989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.536953 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:37.570024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.569988 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca4979b8-86d1-4bc9-b06b-ad36059c1279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.570191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.570031 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca4979b8-86d1-4bc9-b06b-ad36059c1279-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.570191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.570064 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxgv\" (UniqueName: \"kubernetes.io/projected/ca4979b8-86d1-4bc9-b06b-ad36059c1279-kube-api-access-7jxgv\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.570748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.570722 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca4979b8-86d1-4bc9-b06b-ad36059c1279-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.572483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.572453 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca4979b8-86d1-4bc9-b06b-ad36059c1279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.587698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.587671 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxgv\" (UniqueName: \"kubernetes.io/projected/ca4979b8-86d1-4bc9-b06b-ad36059c1279-kube-api-access-7jxgv\") pod \"kuadrant-console-plugin-66c4fb575d-pwnjf\" (UID: \"ca4979b8-86d1-4bc9-b06b-ad36059c1279\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:37.640001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.639923 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:37.724344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:37.724303 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" Oct 30 09:09:38.019983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.019956 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf"] Oct 30 09:09:38.021510 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:09:38.021478 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca4979b8_86d1_4bc9_b06b_ad36059c1279.slice/crio-49f3a524625d1c8fd428e92064ec6f9f5ca8b55586004e933d0406d8e4d9e0d5 WatchSource:0}: Error finding container 49f3a524625d1c8fd428e92064ec6f9f5ca8b55586004e933d0406d8e4d9e0d5: Status 404 returned error can't find the container with id 49f3a524625d1c8fd428e92064ec6f9f5ca8b55586004e933d0406d8e4d9e0d5 Oct 30 09:09:38.023077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.023058 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:09:38.270699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.270615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:38.334456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.334392 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" event={"ID":"ca4979b8-86d1-4bc9-b06b-ad36059c1279","Type":"ContainerStarted","Data":"49f3a524625d1c8fd428e92064ec6f9f5ca8b55586004e933d0406d8e4d9e0d5"} Oct 30 09:09:38.371159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.371126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:38.777631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.777596 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:09:38.781203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.781180 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:38.799267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.799240 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:09:38.831407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.831377 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 09:09:38.860788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.860745 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:09:38.882917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.882876 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tv2c\" (UniqueName: \"kubernetes.io/projected/61f8350c-76c3-464d-9910-c7be862e5628-kube-api-access-5tv2c\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:38.883076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.882924 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61f8350c-76c3-464d-9910-c7be862e5628-config-file\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:38.984131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.984099 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tv2c\" (UniqueName: \"kubernetes.io/projected/61f8350c-76c3-464d-9910-c7be862e5628-kube-api-access-5tv2c\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:38.984359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.984147 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61f8350c-76c3-464d-9910-c7be862e5628-config-file\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:38.984896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:38.984871 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61f8350c-76c3-464d-9910-c7be862e5628-config-file\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:39.001070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:39.001045 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tv2c\" (UniqueName: \"kubernetes.io/projected/61f8350c-76c3-464d-9910-c7be862e5628-kube-api-access-5tv2c\") pod \"limitador-limitador-7c9b87d574-vzh5l\" (UID: \"61f8350c-76c3-464d-9910-c7be862e5628\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:39.098728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:39.098685 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:39.396415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:39.396388 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:09:39.399800 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:09:39.399774 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f8350c_76c3_464d_9910_c7be862e5628.slice/crio-3f5de9a14a42fa1e1f0533a72739649d6c0353ab55a4647c04c79fad3cd56894 WatchSource:0}: Error finding container 3f5de9a14a42fa1e1f0533a72739649d6c0353ab55a4647c04c79fad3cd56894: Status 404 returned error can't find the container with id 3f5de9a14a42fa1e1f0533a72739649d6c0353ab55a4647c04c79fad3cd56894 Oct 30 09:09:40.350755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:40.350701 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" event={"ID":"61f8350c-76c3-464d-9910-c7be862e5628","Type":"ContainerStarted","Data":"e794b6c7c1b26df962241e3d0c934fc079740ae363ab7a6db2ca2c9c5cb2a579"} Oct 30 09:09:40.350944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:40.350811 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" event={"ID":"61f8350c-76c3-464d-9910-c7be862e5628","Type":"ContainerStarted","Data":"3f5de9a14a42fa1e1f0533a72739649d6c0353ab55a4647c04c79fad3cd56894"} Oct 30 09:09:40.350944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:40.350843 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:40.386109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:40.386056 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" podStartSLOduration=1.602281536 podStartE2EDuration="2.386041911s" totalImagesPullingTime="783.760375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:40.383670337 +0000 UTC m=+944.107097505" watchObservedRunningTime="2026-10-30 09:09:40.386041911 +0000 UTC m=+944.109469048" Oct 30 09:09:43.369571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:43.369481 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" event={"ID":"ca4979b8-86d1-4bc9-b06b-ad36059c1279","Type":"ContainerStarted","Data":"b6e2ddf622fb40b4ebea06c24c855b1b16f5f057d98682f420d519929f6a9b54"} Oct 30 09:09:43.402727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:43.402679 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwnjf" podStartSLOduration=1.316921139 podStartE2EDuration="6.402667104s" totalImagesPullingTime="5.085745965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:43.400810959 +0000 UTC m=+947.124238106" watchObservedRunningTime="2026-10-30 09:09:43.402667104 +0000 UTC m=+947.126094247" Oct 30 09:09:47.406164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:47.406118 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:48.013121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:48.013091 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:51.358580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:51.358548 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vzh5l" Oct 30 09:09:53.113399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:53.113366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:53.209433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:53.209400 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:09:53.818009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:09:53.817960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:01.133514 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:10:01.133478 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:10:12.414250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:12.414204 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:12.502893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:12.502854 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:12.590491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:12.590454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:12.683838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:12.683749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:12.760949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:12.760907 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:16.070549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:16.070509 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:16.209147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:16.209113 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:17.838637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:17.838598 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:18.072640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:18.072602 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:31.133924 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:10:31.133878 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:10:32.321295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:32.321258 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:45.107171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:45.107136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:45.223447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:45.223411 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:46.512291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:46.512253 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:46.613504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:46.613471 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:46.783884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:46.783803 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:46.906836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:46.906802 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:47.527054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:47.527021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:49.426303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:49.426262 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:10:56.087922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.087888 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:56.248041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.248009 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:10:56.771241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.771111 4920 generic.go:356] "Generic (PLEG): container finished" podID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerID="83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b" exitCode=0 Oct 30 09:10:56.771241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.771178 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" event={"ID":"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c","Type":"ContainerDied","Data":"83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b"} Oct 30 09:10:56.771601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.771586 4920 scope.go:122] "RemoveContainer" containerID="83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b" Oct 30 09:10:56.772808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.772778 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:10:56.772919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.772874 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:10:56.772984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:56.772921 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:10:57.778677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:57.778644 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" event={"ID":"d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c","Type":"ContainerStarted","Data":"141535bef9aef139495b24940aef8355f626030a59ae457947128ce36ec01063"} Oct 30 09:10:58.049449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:58.049412 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 09:10:58.050604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:58.050581 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:10:58.050675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:58.050655 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 09:10:58.581191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:58.581160 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 09:10:59.048884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:59.048851 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:10:59.049314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:10:59.048909 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 09:11:00.049109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:00.049058 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:11:00.049538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:00.049128 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 09:11:00.393589 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:00.393493 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:00.431786 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:00.431740 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:01.049130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:01.049089 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:11:01.049560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:01.049147 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 09:11:01.146102 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:01.146067 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:11:02.048709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:02.048670 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 09:11:02.048890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:02.048726 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 09:11:03.053232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:03.053184 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 09:11:03.054289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:03.054270 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" Oct 30 09:11:10.443588 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:10.443553 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:15.460538 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:15.460493 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:16.569228 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:16.569174 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:16.569613 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:16.569228 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:20.487408 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:20.487372 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:30.460108 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:30.460068 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:30.499071 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:30.499030 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:31.133566 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:31.133527 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:11:33.662668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:33.662624 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:11:33.750778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:33.750744 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:11:38.058611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:38.058569 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:11:38.156562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:38.156527 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:11:40.548908 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:40.548871 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:45.434888 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:45.434847 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:11:47.656637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:11:47.656600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:11:50.600832 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:11:50.600792 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e849f1_8b29_4f70_a8fc_a9b817a1ed7c.slice/crio-83988e1f367bd45f857398406c66f6e35a3872142ccb18aea77dd9d347d6e86b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:12:01.134973 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:12:01.134928 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:12:02.670969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:02.670920 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:03.183835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:03.183800 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:04.475881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:04.475844 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:04.576124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:04.576083 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:08.931367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:08.931339 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:10.014157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:10.014119 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:12:12.462361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:12.462327 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:13.581869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:13.581835 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:12:14.195775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:14.195740 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:15.390546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.390514 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq"] Oct 30 09:12:15.394268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.394244 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.408550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.408521 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kglgv\"" Oct 30 09:12:15.408697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.408594 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 09:12:15.418724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.418699 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 09:12:15.418905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.418702 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 09:12:15.421273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.421246 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq"] Oct 30 09:12:15.449881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.449851 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.450055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.449892 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.450055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.449955 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.450177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.450145 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab135bfc-97aa-4258-8e8a-5375fa34181d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.450316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.450293 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgfp\" (UniqueName: \"kubernetes.io/projected/ab135bfc-97aa-4258-8e8a-5375fa34181d-kube-api-access-rfgfp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.450381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.450358 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.451421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.451376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:15.550775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550736 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.550958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550807 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab135bfc-97aa-4258-8e8a-5375fa34181d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.550958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550843 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgfp\" (UniqueName: \"kubernetes.io/projected/ab135bfc-97aa-4258-8e8a-5375fa34181d-kube-api-access-rfgfp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.550958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550875 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.550958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550918 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.550958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.550955 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.551409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.551379 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.551500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.551391 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.551500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.551434 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.553093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.553068 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab135bfc-97aa-4258-8e8a-5375fa34181d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.553501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.553482 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab135bfc-97aa-4258-8e8a-5375fa34181d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.574741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.574710 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgfp\" (UniqueName: \"kubernetes.io/projected/ab135bfc-97aa-4258-8e8a-5375fa34181d-kube-api-access-rfgfp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56jtq\" (UID: \"ab135bfc-97aa-4258-8e8a-5375fa34181d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:15.709208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:15.709121 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:16.028684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:16.028589 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq"] Oct 30 09:12:16.034398 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:12:16.034368 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab135bfc_97aa_4258_8e8a_5375fa34181d.slice/crio-5c53dfc7f6ce53326e512bbfe17853c6ab8fe8c19abf64a5f32aeb06a17f891d WatchSource:0}: Error finding container 5c53dfc7f6ce53326e512bbfe17853c6ab8fe8c19abf64a5f32aeb06a17f891d: Status 404 returned error can't find the container with id 5c53dfc7f6ce53326e512bbfe17853c6ab8fe8c19abf64a5f32aeb06a17f891d Oct 30 09:12:16.207869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:16.207832 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" event={"ID":"ab135bfc-97aa-4258-8e8a-5375fa34181d","Type":"ContainerStarted","Data":"5c53dfc7f6ce53326e512bbfe17853c6ab8fe8c19abf64a5f32aeb06a17f891d"} Oct 30 09:12:16.990534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:16.990496 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:17.934105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:17.934068 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:12:18.221035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:18.220943 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" event={"ID":"ab135bfc-97aa-4258-8e8a-5375fa34181d","Type":"ContainerStarted","Data":"9dcfe7ecd3acc6168c58d75898654561cafb8efd14c847d4057a853eeac2bded"} Oct 30 09:12:18.255526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:18.255472 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" podStartSLOduration=2.126388213 podStartE2EDuration="3.25545675s" totalImagesPullingTime="1.129068537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:18.254514128 +0000 UTC m=+1101.977941270" watchObservedRunningTime="2026-10-30 09:12:18.25545675 +0000 UTC m=+1101.978883964" Oct 30 09:12:18.391798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:18.391758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:18.709558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:18.709523 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:19.833304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.833268 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf"] Oct 30 09:12:19.837550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.837524 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.849982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.849951 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 09:12:19.862524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.862495 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf"] Oct 30 09:12:19.877860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.877826 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:19.883159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883131 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.883159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883162 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.883378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883186 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.883378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883339 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.883378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883374 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d12f5f-cf69-485c-af8d-b781fcacf16b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.883479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.883397 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l2qz\" (UniqueName: \"kubernetes.io/projected/81d12f5f-cf69-485c-af8d-b781fcacf16b-kube-api-access-6l2qz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984018 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984075 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d12f5f-cf69-485c-af8d-b781fcacf16b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984108 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2qz\" (UniqueName: \"kubernetes.io/projected/81d12f5f-cf69-485c-af8d-b781fcacf16b-kube-api-access-6l2qz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984151 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984179 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984242 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984504 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984561 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.984714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.984643 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.986489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.986462 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d12f5f-cf69-485c-af8d-b781fcacf16b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:19.986719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:19.986701 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d12f5f-cf69-485c-af8d-b781fcacf16b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:20.000835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:20.000808 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l2qz\" (UniqueName: \"kubernetes.io/projected/81d12f5f-cf69-485c-af8d-b781fcacf16b-kube-api-access-6l2qz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf\" (UID: \"81d12f5f-cf69-485c-af8d-b781fcacf16b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:20.152412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:20.152314 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:20.454006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:20.453975 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf"] Oct 30 09:12:20.455014 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:12:20.454986 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d12f5f_cf69_485c_af8d_b781fcacf16b.slice/crio-de9c453caac99373482dd269fb5c9d02f678e336d1caa5c74f0b13be14b6a216 WatchSource:0}: Error finding container de9c453caac99373482dd269fb5c9d02f678e336d1caa5c74f0b13be14b6a216: Status 404 returned error can't find the container with id de9c453caac99373482dd269fb5c9d02f678e336d1caa5c74f0b13be14b6a216 Oct 30 09:12:21.244193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:21.244161 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" event={"ID":"81d12f5f-cf69-485c-af8d-b781fcacf16b","Type":"ContainerStarted","Data":"f87dcdbb33e9c7ce4948c3af2ae018293d303298907864e15bf4ea1dbe6e485c"} Oct 30 09:12:21.244687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:21.244210 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" event={"ID":"81d12f5f-cf69-485c-af8d-b781fcacf16b","Type":"ContainerStarted","Data":"de9c453caac99373482dd269fb5c9d02f678e336d1caa5c74f0b13be14b6a216"} Oct 30 09:12:21.286983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:21.286915 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" podStartSLOduration=2.105285199 podStartE2EDuration="2.286895905s" totalImagesPullingTime="181.610706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:21.284098569 +0000 UTC m=+1105.007525734" watchObservedRunningTime="2026-10-30 09:12:21.286895905 +0000 UTC m=+1105.010323049" Oct 30 09:12:22.154113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.154073 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n"] Oct 30 09:12:22.159178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.159150 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:22.159404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.159284 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.175945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.175908 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n"] Oct 30 09:12:22.182313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.182280 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 09:12:22.209696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209646 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.209900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209723 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.209900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209759 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.209900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209861 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e36101-1742-43d6-82dd-010f3d70bcf1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.210098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209900 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsljn\" (UniqueName: \"kubernetes.io/projected/46e36101-1742-43d6-82dd-010f3d70bcf1-kube-api-access-jsljn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.210098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.209960 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.213892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.213855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:22.310884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.310839 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e36101-1742-43d6-82dd-010f3d70bcf1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.311432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.310895 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsljn\" (UniqueName: \"kubernetes.io/projected/46e36101-1742-43d6-82dd-010f3d70bcf1-kube-api-access-jsljn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.311432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.310960 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.311432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.311027 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.311432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.311131 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.311432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.311170 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.312359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.312326 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.312480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.312128 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.312567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.312539 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.314339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.314312 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e36101-1742-43d6-82dd-010f3d70bcf1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.314704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.314678 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e36101-1742-43d6-82dd-010f3d70bcf1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.337258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.337183 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsljn\" (UniqueName: \"kubernetes.io/projected/46e36101-1742-43d6-82dd-010f3d70bcf1-kube-api-access-jsljn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n\" (UID: \"46e36101-1742-43d6-82dd-010f3d70bcf1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.474869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.474783 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:22.774872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:22.774844 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n"] Oct 30 09:12:22.776442 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:12:22.776413 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e36101_1742_43d6_82dd_010f3d70bcf1.slice/crio-71ed33d313f35d2c33bb45efff9ef87b60dd0d4b6bec3e81de704ee2725dc1a4 WatchSource:0}: Error finding container 71ed33d313f35d2c33bb45efff9ef87b60dd0d4b6bec3e81de704ee2725dc1a4: Status 404 returned error can't find the container with id 71ed33d313f35d2c33bb45efff9ef87b60dd0d4b6bec3e81de704ee2725dc1a4 Oct 30 09:12:23.261655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:23.261613 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" event={"ID":"46e36101-1742-43d6-82dd-010f3d70bcf1","Type":"ContainerStarted","Data":"3064bdf869b1a18eac762cfed3377f81ab48c5a587148efd2a6979c251e25920"} Oct 30 09:12:23.261655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:23.261658 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" event={"ID":"46e36101-1742-43d6-82dd-010f3d70bcf1","Type":"ContainerStarted","Data":"71ed33d313f35d2c33bb45efff9ef87b60dd0d4b6bec3e81de704ee2725dc1a4"} Oct 30 09:12:23.475850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:23.475813 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:23.954491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:23.954431 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podStartSLOduration=1.7430385419999999 podStartE2EDuration="1.954409626s" totalImagesPullingTime="211.371084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:23.295430953 +0000 UTC m=+1107.018858115" watchObservedRunningTime="2026-10-30 09:12:23.954409626 +0000 UTC m=+1107.677836769" Oct 30 09:12:23.955689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:23.955672 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:12:25.709391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:25.709354 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:25.728533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:25.728501 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:26.289902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:26.289875 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56jtq" Oct 30 09:12:30.152892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:30.152855 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:30.165078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:30.165051 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:30.310199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:30.310165 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf" Oct 30 09:12:31.134732 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:12:31.134697 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:12:32.475483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:32.475447 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:32.488247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:32.488200 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:33.324826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:33.324793 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:12:34.884390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.884357 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:12:34.888309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.888287 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.919689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.919657 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:12:34.927440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927408 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927442 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927465 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927571 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927602 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927623 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927735 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927772 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:34.927895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:34.927792 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkgz\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028663 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028710 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028737 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028757 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028777 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028803 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028833 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028859 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.028982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.028886 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkgz\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.029389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.029136 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.029733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.029703 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.030088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.030066 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.030259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.030208 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.030416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.030392 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.031516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.031498 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.031677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.031657 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.046744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.046714 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.046909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.046755 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkgz\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\") pod \"maas-default-gateway-openshift-default-77f556d647-kg5zv\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.203002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.202895 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:35.523285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.523256 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:12:35.524896 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:12:35.524865 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3660829a_32df_4c3f_bbe7_af4ce2930250.slice/crio-880c88a1d031d41b0c53f850927d4bc59d830868df587b5343216d0de526ac97 WatchSource:0}: Error finding container 880c88a1d031d41b0c53f850927d4bc59d830868df587b5343216d0de526ac97: Status 404 returned error can't find the container with id 880c88a1d031d41b0c53f850927d4bc59d830868df587b5343216d0de526ac97 Oct 30 09:12:35.527143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.527111 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:35.527249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.527180 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:35.527249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:35.527207 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:36.333968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:36.333927 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" event={"ID":"3660829a-32df-4c3f-bbe7-af4ce2930250","Type":"ContainerStarted","Data":"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45"} Oct 30 09:12:36.334444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:36.333982 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" event={"ID":"3660829a-32df-4c3f-bbe7-af4ce2930250","Type":"ContainerStarted","Data":"880c88a1d031d41b0c53f850927d4bc59d830868df587b5343216d0de526ac97"} Oct 30 09:12:36.391330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:36.391274 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podStartSLOduration=2.391258504 podStartE2EDuration="2.391258504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:36.379708966 +0000 UTC m=+1120.103136129" watchObservedRunningTime="2026-10-30 09:12:36.391258504 +0000 UTC m=+1120.114685650" Oct 30 09:12:37.211630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:37.211590 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:37.217493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:37.217459 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:37.217772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:37.217738 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:38.204707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:38.204664 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:38.205353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:38.205317 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:39.204241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:39.203651 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:39.204241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:39.203752 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:39.737065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:39.737030 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:39.837967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:39.837931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:40.213241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:40.206601 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:40.213241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:40.206675 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:41.209239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:41.208872 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:41.209239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:41.208949 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:41.343834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:41.343799 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:41.443449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:41.443417 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:42.203440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:42.203407 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 09:12:42.203604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:42.203466 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 09:12:43.807548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:43.807503 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:43.808012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:43.807967 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:43.808832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:43.808810 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:12:47.762764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:47.762727 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:12:49.931911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:49.931870 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:12:49.936958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:49.936938 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:49.958506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:49.958472 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:12:50.078187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078132 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078275 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078313 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078396 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078472 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078555 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gbg\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078608 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078655 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.078722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.078708 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.179880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.179840 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gbg\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.179900 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.179931 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.179962 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.179989 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180038 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180067 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180109 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180150 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180442 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180490 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180520 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.180649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.180610 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.181041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.181016 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.182489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.182447 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.182538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.182515 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.195987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.195966 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.196279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.196262 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gbg\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\") pod \"maas-default-gateway-openshift-default-77f556d647-5krd5\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.251375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.251344 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:50.561980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.561955 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:12:50.564332 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:12:50.564295 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dda1e71_7a9c_4734_929b_f39c8c72a42d.slice/crio-694a47b85874d58b949386f35dbade0463faf77c8f8b8e4182702b2ae6417e9a WatchSource:0}: Error finding container 694a47b85874d58b949386f35dbade0463faf77c8f8b8e4182702b2ae6417e9a: Status 404 returned error can't find the container with id 694a47b85874d58b949386f35dbade0463faf77c8f8b8e4182702b2ae6417e9a Oct 30 09:12:50.566518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.566489 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:50.566605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.566555 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:50.566605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:50.566584 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:51.429139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:51.429100 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" event={"ID":"1dda1e71-7a9c-4734-929b-f39c8c72a42d","Type":"ContainerStarted","Data":"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8"} Oct 30 09:12:51.429532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:51.429154 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" event={"ID":"1dda1e71-7a9c-4734-929b-f39c8c72a42d","Type":"ContainerStarted","Data":"694a47b85874d58b949386f35dbade0463faf77c8f8b8e4182702b2ae6417e9a"} Oct 30 09:12:51.479033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:51.478974 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podStartSLOduration=2.478958765 podStartE2EDuration="2.478958765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:51.477009966 +0000 UTC m=+1135.200437120" watchObservedRunningTime="2026-10-30 09:12:51.478958765 +0000 UTC m=+1135.202385907" Oct 30 09:12:52.252237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:52.252176 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:52.253754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:52.253725 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:52.253873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:52.253781 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:53.252042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:53.252004 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:53.252487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:53.252061 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:54.252334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:54.252299 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:54.252747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:54.252352 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:55.251920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:55.251883 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:55.252131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:55.251947 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:56.252737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:56.252700 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:56.253177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:56.252753 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:57.252113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:57.252073 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5krd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 09:12:57.252317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:57.252130 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 09:12:58.889612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:58.889579 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:58.890117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:58.890008 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:58.891000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:58.890979 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:12:59.075599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:12:59.075564 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:01.132894 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:01.132857 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:13:07.453393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:07.453356 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:07.553592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:07.553555 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:07.657609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:07.657558 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:07.764070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:07.763988 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.039093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.039059 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.143827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.143794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.201274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.201237 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:08.207469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.207440 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.222684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.222555 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ffc2a58-openshift-default-dockercfg-4czfp\"" Oct 30 09:13:08.236642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.236606 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:08.244527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.244497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.272334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272304 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272338 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272437 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272469 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272546 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272564 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272618 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272642 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.272801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.272675 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksfb\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.347586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.347490 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.373822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373784 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.373822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373821 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373844 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373866 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373896 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksfb\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373928 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.373947 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374018 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374443 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374454 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374495 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374510 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374640 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.374802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.374788 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.376382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.376357 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.376589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.376572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.393375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.393340 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.393779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.393756 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksfb\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\") pod \"e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.460064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.460030 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.525120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.525092 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:08.707570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.707465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:08.873188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.873164 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:08.875806 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:08.875780 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8596595e_3cb5_4dce_aa66_0a507f1e6c23.slice/crio-48658854ec79ac5ed5a763ef30dc7b14c827e00bf92c1a45d0a7e7344ef2e8e9 WatchSource:0}: Error finding container 48658854ec79ac5ed5a763ef30dc7b14c827e00bf92c1a45d0a7e7344ef2e8e9: Status 404 returned error can't find the container with id 48658854ec79ac5ed5a763ef30dc7b14c827e00bf92c1a45d0a7e7344ef2e8e9 Oct 30 09:13:08.878114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.878083 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.878186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.878148 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.878186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.878176 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.954295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:08.954265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:09.474990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:09.474957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:09.530297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:09.530249 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerStarted","Data":"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059"} Oct 30 09:13:09.530488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:09.530326 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerStarted","Data":"48658854ec79ac5ed5a763ef30dc7b14c827e00bf92c1a45d0a7e7344ef2e8e9"} Oct 30 09:13:09.576816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:09.576767 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podStartSLOduration=1.576752588 podStartE2EDuration="1.576752588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:09.576098548 +0000 UTC m=+1153.299525702" watchObservedRunningTime="2026-10-30 09:13:09.576752588 +0000 UTC m=+1153.300179731" Oct 30 09:13:09.929572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:09.929537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:10.028706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.028667 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:10.526064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.526033 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:10.531629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.531603 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:10.536018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.535986 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:10.536727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.536709 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:10.884256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.884208 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:10.984631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:10.984591 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:11.185859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:11.185783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:11.285341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:11.285303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:12.581835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.581806 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:12.678955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.678911 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:12.766211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.766179 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:12.770008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.769990 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.786296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.786272 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:12.793478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.793452 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-70374d4e\"" Oct 30 09:13:12.824864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.824822 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfzh\" (UniqueName: \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.825050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.825023 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.858388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.858305 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-70374d4e-dockercfg-vwk5c\"" Oct 30 09:13:12.875506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.875467 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:12.880303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.880273 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:12.881646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.881607 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:12.926064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.926023 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.926284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.926076 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9l6\" (UniqueName: \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:12.926284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.926140 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfzh\" (UniqueName: \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.926284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.926171 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:12.926909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.926887 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:12.942911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:12.942876 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfzh\" (UniqueName: \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\") pod \"payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:13.026885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.026846 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:13.027094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.026979 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9l6\" (UniqueName: \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:13.027642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.027618 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:13.044000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.043970 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9l6\" (UniqueName: \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\") pod \"payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:13.083993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.083955 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:13.197870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.197834 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:13.219030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.218977 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:13.319799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.319763 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:13.386029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.386002 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:13.387017 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:13.386994 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fd7d94_a7c2_4855_8256_365af8b51408.slice/crio-0a84aba5bd1231b0d14ece7664ad13fb903467d49cc490b2b37e1e3e4689b596 WatchSource:0}: Error finding container 0a84aba5bd1231b0d14ece7664ad13fb903467d49cc490b2b37e1e3e4689b596: Status 404 returned error can't find the container with id 0a84aba5bd1231b0d14ece7664ad13fb903467d49cc490b2b37e1e3e4689b596 Oct 30 09:13:13.494069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.494043 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:13.495188 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:13.495158 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63a8edd_828b_4696_b88e_f0a384ec1fb5.slice/crio-a337dfd9bea2116eed864a4d9fcdfc009e0e6f1b16f25b48f2d46eef6a4881d6 WatchSource:0}: Error finding container a337dfd9bea2116eed864a4d9fcdfc009e0e6f1b16f25b48f2d46eef6a4881d6: Status 404 returned error can't find the container with id a337dfd9bea2116eed864a4d9fcdfc009e0e6f1b16f25b48f2d46eef6a4881d6 Oct 30 09:13:13.554600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.554566 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" event={"ID":"d63a8edd-828b-4696-b88e-f0a384ec1fb5","Type":"ContainerStarted","Data":"a337dfd9bea2116eed864a4d9fcdfc009e0e6f1b16f25b48f2d46eef6a4881d6"} Oct 30 09:13:13.555912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.555877 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" event={"ID":"15fd7d94-a7c2-4855-8256-365af8b51408","Type":"ContainerStarted","Data":"0a84aba5bd1231b0d14ece7664ad13fb903467d49cc490b2b37e1e3e4689b596"} Oct 30 09:13:13.997508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:13.997418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:14.106691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.106652 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:14.203404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.203362 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:14.305570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.305540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:14.501645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.501606 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:14.524842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.524803 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:14.529319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.529292 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.547521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.547497 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:14.554074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.554040 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ffc2a58\"" Oct 30 09:13:14.594191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.594107 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:14.602170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.601955 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.614778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.614690 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:14.616868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.616571 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2ffc2a58-dockercfg-gjgs5\"" Oct 30 09:13:14.645291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.645248 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.645496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.645317 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.645496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.645365 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwh8c\" (UniqueName: \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.645496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.645447 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk74k\" (UniqueName: \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.746180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.746139 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.746373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.746262 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.746373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.746312 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwh8c\" (UniqueName: \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.746373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.746328 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk74k\" (UniqueName: \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.747353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.747306 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.748072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.748038 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.764058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.764026 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk74k\" (UniqueName: \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\") pod \"payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:14.768852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.768815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwh8c\" (UniqueName: \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\") pod \"payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.848136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.847835 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:14.921775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:14.919970 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:15.042304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.042271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.141877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.141791 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.167916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.167251 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:15.188549 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:15.180918 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode688bbf5_b10e_49aa_9f4c_462467e013ba.slice/crio-8aa4af75558e240ce4d972ee9fd84a410e07c93322417c9d846802874b2dbb7c WatchSource:0}: Error finding container 8aa4af75558e240ce4d972ee9fd84a410e07c93322417c9d846802874b2dbb7c: Status 404 returned error can't find the container with id 8aa4af75558e240ce4d972ee9fd84a410e07c93322417c9d846802874b2dbb7c Oct 30 09:13:15.257099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.257028 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.265149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.265117 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:15.269194 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:15.269162 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2cf9ac_abe6_4dd0_8c51_a83b63917284.slice/crio-94e77e37a38862e92a705d372274b87270b937466ff69302bf994b9ad6227d67 WatchSource:0}: Error finding container 94e77e37a38862e92a705d372274b87270b937466ff69302bf994b9ad6227d67: Status 404 returned error can't find the container with id 94e77e37a38862e92a705d372274b87270b937466ff69302bf994b9ad6227d67 Oct 30 09:13:15.356933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.356895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.450955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.450879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.452984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.451139 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:15.465290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.465267 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.475297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.475270 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:15.482018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.481973 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6233a0dd-dockercfg-pwvn8\"" Oct 30 09:13:15.492092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.492068 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6233a0dd\"" Oct 30 09:13:15.551046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.544580 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:15.551046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.550006 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.557758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.557726 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2c9\" (UniqueName: \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.557915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.557881 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.567910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.567883 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:15.575933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.575894 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" event={"ID":"e688bbf5-b10e-49aa-9f4c-462467e013ba","Type":"ContainerStarted","Data":"8aa4af75558e240ce4d972ee9fd84a410e07c93322417c9d846802874b2dbb7c"} Oct 30 09:13:15.577696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.577637 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" event={"ID":"5c2cf9ac-abe6-4dd0-8c51-a83b63917284","Type":"ContainerStarted","Data":"94e77e37a38862e92a705d372274b87270b937466ff69302bf994b9ad6227d67"} Oct 30 09:13:15.658810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.658771 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.659067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.659049 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s6m\" (UniqueName: \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.659142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.659114 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2c9\" (UniqueName: \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.659209 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.659185 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.660124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.660099 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.680192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.680081 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2c9\" (UniqueName: \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\") pod \"payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.766627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.766543 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.766816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.766689 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s6m\" (UniqueName: \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.767803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.767746 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.788147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.788097 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:15.789396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.789367 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s6m\" (UniqueName: \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\") pod \"payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.876327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.871885 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:15.876534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.876454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:15.972529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:15.972440 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:16.056974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.056872 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:16.122883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.120437 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:16.175715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.175649 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:16.240084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.240002 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:16.242576 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:16.242540 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c1c128_233a_4edb_913f_cf97b4d138b9.slice/crio-c0b6c498448e9324d1dec50cb5e9fd3eea5e31210dfe0e9ef6555da1d4aa03a2 WatchSource:0}: Error finding container c0b6c498448e9324d1dec50cb5e9fd3eea5e31210dfe0e9ef6555da1d4aa03a2: Status 404 returned error can't find the container with id c0b6c498448e9324d1dec50cb5e9fd3eea5e31210dfe0e9ef6555da1d4aa03a2 Oct 30 09:13:16.258722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.258694 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:16.592592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.592548 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" event={"ID":"08c1c128-233a-4edb-913f-cf97b4d138b9","Type":"ContainerStarted","Data":"c0b6c498448e9324d1dec50cb5e9fd3eea5e31210dfe0e9ef6555da1d4aa03a2"} Oct 30 09:13:16.594776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.594743 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" event={"ID":"239411df-a827-4427-bd9b-9210b81781b9","Type":"ContainerStarted","Data":"4b6d74821c2b4d1bdf6f63489ff4d03fb16871fb62d04b1d1e7f0172506509e6"} Oct 30 09:13:16.999539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:16.999444 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:17.098077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.097961 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:17.187617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.185845 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:17.198388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.198364 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.207990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.207962 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:17.216917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.216890 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c806f9c\"" Oct 30 09:13:17.217070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.217048 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c806f9c-dockercfg-s2mq8\"" Oct 30 09:13:17.284288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.282706 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:17.309274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.294066 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.309274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.302186 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:17.313959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.313922 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587qs\" (UniqueName: \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.314128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.314068 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.415395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.415352 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.415632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.415457 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587qs\" (UniqueName: \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.415632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.415487 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctvn\" (UniqueName: \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.415632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.415557 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.416768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.416489 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.445257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.437322 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-587qs\" (UniqueName: \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\") pod \"payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.517477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.516941 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctvn\" (UniqueName: \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.517477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.517026 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.518006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.517977 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.520640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.520471 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:17.539103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.538987 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctvn\" (UniqueName: \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\") pod \"payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.630466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.629827 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:17.755191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.755116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:17.859942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.859837 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:17.877376 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:17.877338 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda066df90_b698_483b_9b74_3f0f0f6fa43a.slice/crio-7388c110cc362fcb561892eee7f2ce9d6206e69ecf0433b2664d29bb15949e71 WatchSource:0}: Error finding container 7388c110cc362fcb561892eee7f2ce9d6206e69ecf0433b2664d29bb15949e71: Status 404 returned error can't find the container with id 7388c110cc362fcb561892eee7f2ce9d6206e69ecf0433b2664d29bb15949e71 Oct 30 09:13:17.881959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.881932 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:17.978250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.977817 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:17.982886 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:17.982583 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa8f70d_9cda_4abb_951c_ba6efb988a3e.slice/crio-743904545ea88dccb0931f3fd6ce9dd89750ce4489e41a743a1e8d5cec934f9b WatchSource:0}: Error finding container 743904545ea88dccb0931f3fd6ce9dd89750ce4489e41a743a1e8d5cec934f9b: Status 404 returned error can't find the container with id 743904545ea88dccb0931f3fd6ce9dd89750ce4489e41a743a1e8d5cec934f9b Oct 30 09:13:17.984735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:17.984293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:18.616180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.616073 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" event={"ID":"e688bbf5-b10e-49aa-9f4c-462467e013ba","Type":"ContainerStarted","Data":"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab"} Oct 30 09:13:18.616664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.616285 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:18.618013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.617982 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" event={"ID":"239411df-a827-4427-bd9b-9210b81781b9","Type":"ContainerStarted","Data":"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9"} Oct 30 09:13:18.618131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.618068 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:18.619793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.619770 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" event={"ID":"d63a8edd-828b-4696-b88e-f0a384ec1fb5","Type":"ContainerStarted","Data":"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3"} Oct 30 09:13:18.619888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.619835 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:18.621352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.621330 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" event={"ID":"5c2cf9ac-abe6-4dd0-8c51-a83b63917284","Type":"ContainerStarted","Data":"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10"} Oct 30 09:13:18.621434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.621414 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:18.622860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.622839 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" event={"ID":"a066df90-b698-483b-9b74-3f0f0f6fa43a","Type":"ContainerStarted","Data":"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291"} Oct 30 09:13:18.622953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.622879 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" event={"ID":"a066df90-b698-483b-9b74-3f0f0f6fa43a","Type":"ContainerStarted","Data":"7388c110cc362fcb561892eee7f2ce9d6206e69ecf0433b2664d29bb15949e71"} Oct 30 09:13:18.622953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.622905 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:18.624346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.624326 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" event={"ID":"08c1c128-233a-4edb-913f-cf97b4d138b9","Type":"ContainerStarted","Data":"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf"} Oct 30 09:13:18.624443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.624373 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:18.625813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.625792 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" event={"ID":"15fd7d94-a7c2-4855-8256-365af8b51408","Type":"ContainerStarted","Data":"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f"} Oct 30 09:13:18.626540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.625842 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:18.627285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.627266 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" event={"ID":"0fa8f70d-9cda-4abb-951c-ba6efb988a3e","Type":"ContainerStarted","Data":"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e"} Oct 30 09:13:18.627352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.627304 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" event={"ID":"0fa8f70d-9cda-4abb-951c-ba6efb988a3e","Type":"ContainerStarted","Data":"743904545ea88dccb0931f3fd6ce9dd89750ce4489e41a743a1e8d5cec934f9b"} Oct 30 09:13:18.627402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.627388 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:18.644183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.644153 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:13:18.651503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.651459 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" podStartSLOduration=2.089530345 podStartE2EDuration="4.651445143s" totalImagesPullingTime="2.561914798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.649618018 +0000 UTC m=+1162.373045163" watchObservedRunningTime="2026-10-30 09:13:18.651445143 +0000 UTC m=+1162.374872286" Oct 30 09:13:18.723968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.723909 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" podStartSLOduration=1.719878246 podStartE2EDuration="3.723893779s" totalImagesPullingTime="2.004015533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.721979669 +0000 UTC m=+1162.445406812" watchObservedRunningTime="2026-10-30 09:13:18.723893779 +0000 UTC m=+1162.447320922" Oct 30 09:13:18.765075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.765013 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" podStartSLOduration=1.764998418 podStartE2EDuration="1.764998418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.76391364 +0000 UTC m=+1162.487340811" watchObservedRunningTime="2026-10-30 09:13:18.764998418 +0000 UTC m=+1162.488425558" Oct 30 09:13:18.811389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.811337 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" podStartSLOduration=1.811322358 podStartE2EDuration="1.811322358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.810047756 +0000 UTC m=+1162.533474898" watchObservedRunningTime="2026-10-30 09:13:18.811322358 +0000 UTC m=+1162.534749501" Oct 30 09:13:18.845821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.845764 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" podStartSLOduration=2.641201225 podStartE2EDuration="6.84574531s" totalImagesPullingTime="4.204544085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.844910085 +0000 UTC m=+1162.568337255" watchObservedRunningTime="2026-10-30 09:13:18.84574531 +0000 UTC m=+1162.569172452" Oct 30 09:13:18.880520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.880411 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" podStartSLOduration=2.860099 podStartE2EDuration="6.880396752s" totalImagesPullingTime="4.020297752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.878909653 +0000 UTC m=+1162.602336796" watchObservedRunningTime="2026-10-30 09:13:18.880396752 +0000 UTC m=+1162.603823894" Oct 30 09:13:18.914534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.914458 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" podStartSLOduration=1.747955257 podStartE2EDuration="3.914443248s" totalImagesPullingTime="2.166487991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.912879432 +0000 UTC m=+1162.636306577" watchObservedRunningTime="2026-10-30 09:13:18.914443248 +0000 UTC m=+1162.637870394" Oct 30 09:13:18.946372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:18.946325 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" podStartSLOduration=2.492286923 podStartE2EDuration="4.946312111s" totalImagesPullingTime="2.454025188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:18.946003502 +0000 UTC m=+1162.669430647" watchObservedRunningTime="2026-10-30 09:13:18.946312111 +0000 UTC m=+1162.669739537" Oct 30 09:13:19.017796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:19.017746 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:19.122603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:19.122565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:19.680612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:19.680569 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:20.932466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:20.932436 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:21.030076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.030045 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:21.125808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.125763 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:13:21.130526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.130497 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.144242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.144197 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:13:21.151106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.151085 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6cf9ebb1\"" Oct 30 09:13:21.151305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.151288 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6cf9ebb1-dockercfg-5sm87\"" Oct 30 09:13:21.262685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.262590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46hl\" (UniqueName: \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.262685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.262666 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.363926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.363891 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46hl\" (UniqueName: \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.364123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.363970 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.364663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.364644 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.371716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.371687 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:21.381160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.381133 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46hl\" (UniqueName: \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\") pod \"payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.444883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.444844 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:21.453825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.453792 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:21.550039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.549972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:21.775282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:21.775254 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:13:21.776351 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:21.776325 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c97384_406a_4027_a3fd_6db6d1c1fc64.slice/crio-39a420644a6b391605af392236cf50fb6489796214543c296af7bff3aa039544 WatchSource:0}: Error finding container 39a420644a6b391605af392236cf50fb6489796214543c296af7bff3aa039544: Status 404 returned error can't find the container with id 39a420644a6b391605af392236cf50fb6489796214543c296af7bff3aa039544 Oct 30 09:13:22.654082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:22.654043 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" event={"ID":"e0c97384-406a-4027-a3fd-6db6d1c1fc64","Type":"ContainerStarted","Data":"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b"} Oct 30 09:13:22.654558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:22.654106 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" event={"ID":"e0c97384-406a-4027-a3fd-6db6d1c1fc64","Type":"ContainerStarted","Data":"39a420644a6b391605af392236cf50fb6489796214543c296af7bff3aa039544"} Oct 30 09:13:22.654558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:22.654257 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:22.688889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:22.688832 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" podStartSLOduration=1.688817241 podStartE2EDuration="1.688817241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:22.686694994 +0000 UTC m=+1166.410122135" watchObservedRunningTime="2026-10-30 09:13:22.688817241 +0000 UTC m=+1166.412244397" Oct 30 09:13:22.979937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:22.979859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:24.046038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:24.045994 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:13:24.765992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:24.765947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:26.661882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.661851 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:26.762805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.762773 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:26.862933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.862898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:26.963714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.963637 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw"] Oct 30 09:13:26.970350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.970310 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:26.983346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.983312 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw"] Oct 30 09:13:26.989470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.989424 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:13:26.989790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.989753 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:13:26.990044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.990025 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:13:26.994004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:26.993960 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:13:27.046692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.046652 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.046909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.046719 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twfn\" (UniqueName: \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.046909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.046857 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.147956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.147913 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.148182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.147991 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.148182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.148112 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twfn\" (UniqueName: \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.148850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.148821 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.150422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.150393 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.166294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.166265 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twfn\" (UniqueName: \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\") pod \"maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.295450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.295416 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:27.597547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.597516 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw"] Oct 30 09:13:27.598175 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:27.598148 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d8ede7_39da_4cc3_9ef2_302dddde51ea.slice/crio-6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763 WatchSource:0}: Error finding container 6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763: Status 404 returned error can't find the container with id 6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763 Oct 30 09:13:27.685481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:27.685436 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" event={"ID":"29d8ede7-39da-4cc3-9ef2-302dddde51ea","Type":"ContainerStarted","Data":"6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763"} Oct 30 09:13:28.577409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:28.577369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:28.988780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:28.988707 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t"] Oct 30 09:13:28.994475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:28.994450 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.012354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.011679 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t"] Oct 30 09:13:29.071081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.071045 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9sq\" (UniqueName: \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.073120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.071096 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.073120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.071237 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.172538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.172501 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9sq\" (UniqueName: \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.172681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.172643 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.172735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.172695 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.173765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.173535 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.175837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.175767 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.189924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.189901 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9sq\" (UniqueName: \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\") pod \"maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.312161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.312126 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:29.633865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.633795 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t"] Oct 30 09:13:29.636993 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:29.636945 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907b0422_9d7d_477f_8d1a_af63f9236529.slice/crio-bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341 WatchSource:0}: Error finding container bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341: Status 404 returned error can't find the container with id bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341 Oct 30 09:13:29.637233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.637189 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:29.637358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.637309 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:29.637749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.637720 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:29.639070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.638442 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:29.639397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.639378 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:29.639490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.639469 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:29.639836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.639807 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:29.639984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.639965 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:29.704156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.704119 4920 generic.go:356] "Generic (PLEG): container finished" podID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerID="149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059" exitCode=0 Oct 30 09:13:29.704345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.704309 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerDied","Data":"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059"} Oct 30 09:13:29.704808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.704786 4920 scope.go:122] "RemoveContainer" containerID="149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059" Oct 30 09:13:29.708057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.708015 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:29.708188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.708120 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:29.708188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.708175 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:29.709666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:29.709610 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" event={"ID":"907b0422-9d7d-477f-8d1a-af63f9236529","Type":"ContainerStarted","Data":"bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341"} Oct 30 09:13:30.556999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:30.556954 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:30.731059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:30.731017 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" event={"ID":"29d8ede7-39da-4cc3-9ef2-302dddde51ea","Type":"ContainerStarted","Data":"4fb8c18303603dda2da4c047e5491047686a46797ddd1ca94079c86704e701b2"} Oct 30 09:13:30.735621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:30.735583 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerStarted","Data":"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36"} Oct 30 09:13:30.766351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:30.766300 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" podStartSLOduration=2.149149853 podStartE2EDuration="4.76628604s" totalImagesPullingTime="2.617136187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:30.766020703 +0000 UTC m=+1174.489447846" watchObservedRunningTime="2026-10-30 09:13:30.76628604 +0000 UTC m=+1174.489713183" Oct 30 09:13:31.158354 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:31.158243 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:13:31.526143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.526097 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:31.526539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.526516 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:13:31.526634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.526567 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 09:13:31.742593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.742501 4920 generic.go:356] "Generic (PLEG): container finished" podID="29d8ede7-39da-4cc3-9ef2-302dddde51ea" containerID="4fb8c18303603dda2da4c047e5491047686a46797ddd1ca94079c86704e701b2" exitCode=0 Oct 30 09:13:31.743023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.742592 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" event={"ID":"29d8ede7-39da-4cc3-9ef2-302dddde51ea","Type":"ContainerDied","Data":"4fb8c18303603dda2da4c047e5491047686a46797ddd1ca94079c86704e701b2"} Oct 30 09:13:31.744302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.744276 4920 generic.go:356] "Generic (PLEG): container finished" podID="907b0422-9d7d-477f-8d1a-af63f9236529" containerID="faff55648fc419c09d2e9c6122e23a95b49c15987b5e07d42f58ac7cae20997a" exitCode=0 Oct 30 09:13:31.744427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:31.744354 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" event={"ID":"907b0422-9d7d-477f-8d1a-af63f9236529","Type":"ContainerDied","Data":"faff55648fc419c09d2e9c6122e23a95b49c15987b5e07d42f58ac7cae20997a"} Oct 30 09:13:32.526317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:32.526281 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:13:32.526503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:32.526338 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 09:13:32.783061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:32.782960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:32.964438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:32.964409 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:33.023250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.011638 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:33.053792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.053759 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\" (UniqueName: \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\") pod \"907b0422-9d7d-477f-8d1a-af63f9236529\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " Oct 30 09:13:33.053980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.053807 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\" (UniqueName: \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\") pod \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " Oct 30 09:13:33.053980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.053865 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\") pod \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " Oct 30 09:13:33.053980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.053920 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\") pod \"907b0422-9d7d-477f-8d1a-af63f9236529\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " Oct 30 09:13:33.053980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.053973 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\") pod \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\" (UID: \"29d8ede7-39da-4cc3-9ef2-302dddde51ea\") " Oct 30 09:13:33.054210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.054028 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\") pod \"907b0422-9d7d-477f-8d1a-af63f9236529\" (UID: \"907b0422-9d7d-477f-8d1a-af63f9236529\") " Oct 30 09:13:33.054533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.054506 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca" pod "29d8ede7-39da-4cc3-9ef2-302dddde51ea" (UID: "29d8ede7-39da-4cc3-9ef2-302dddde51ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:33.055488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.055372 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca" pod "907b0422-9d7d-477f-8d1a-af63f9236529" (UID: "907b0422-9d7d-477f-8d1a-af63f9236529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:33.057247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.057197 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca" pod "907b0422-9d7d-477f-8d1a-af63f9236529" (UID: "907b0422-9d7d-477f-8d1a-af63f9236529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:33.057523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.057491 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca" pod "29d8ede7-39da-4cc3-9ef2-302dddde51ea" (UID: "29d8ede7-39da-4cc3-9ef2-302dddde51ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:33.058766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.057970 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq" pod "907b0422-9d7d-477f-8d1a-af63f9236529" (UID: "907b0422-9d7d-477f-8d1a-af63f9236529"). InnerVolumeSpecName "kube-api-access-9b9sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:33.059091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.059074 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn" pod "29d8ede7-39da-4cc3-9ef2-302dddde51ea" (UID: "29d8ede7-39da-4cc3-9ef2-302dddde51ea"). InnerVolumeSpecName "kube-api-access-7twfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:33.155122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155078 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9sq\" (UniqueName: \"kubernetes.io/projected/907b0422-9d7d-477f-8d1a-af63f9236529-kube-api-access-9b9sq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.155122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155116 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twfn\" (UniqueName: \"kubernetes.io/projected/29d8ede7-39da-4cc3-9ef2-302dddde51ea-kube-api-access-7twfn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.155122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155127 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.155471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155136 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.155471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155146 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29d8ede7-39da-4cc3-9ef2-302dddde51ea-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.155471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.155155 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907b0422-9d7d-477f-8d1a-af63f9236529-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:33.526164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.526067 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:13:33.526164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.526132 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 09:13:33.663263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.663235 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:13:33.767313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.767285 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" Oct 30 09:13:33.767479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.767321 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t" event={"ID":"907b0422-9d7d-477f-8d1a-af63f9236529","Type":"ContainerDied","Data":"bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341"} Oct 30 09:13:33.767479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.767393 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbba69071dd78f6dcc3782cf4c85ee279f6aee1263d96dd9bae816cf1442b341" Oct 30 09:13:33.768887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.768866 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" event={"ID":"29d8ede7-39da-4cc3-9ef2-302dddde51ea","Type":"ContainerDied","Data":"6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763"} Oct 30 09:13:33.768887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.768889 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw" Oct 30 09:13:33.769053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:33.768899 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d10b9fd6d8f0d347644c5d0823594077ef630242571e49f751db4c445fe5763" Oct 30 09:13:34.525727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:34.525687 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:13:34.528372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:34.528332 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 09:13:34.782664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:34.782514 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:34.888606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:34.888566 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:13:35.526427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:35.526388 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:13:35.526962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:35.526454 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 09:13:36.529756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:36.529720 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:36.530194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:36.530121 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:36.530896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:36.530877 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:37.046122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.046088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:37.296905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.296813 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:37.331647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.331614 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:37.397227 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.397178 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx"] Oct 30 09:13:37.397990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.397955 4920 cpu_manager.go:414] "containerMap: removing container" podUID="29d8ede7-39da-4cc3-9ef2-302dddde51ea" containerName="revoke-keys" Oct 30 09:13:37.397990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.397980 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d8ede7-39da-4cc3-9ef2-302dddde51ea" containerName="revoke-keys" Oct 30 09:13:37.397990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.397994 4920 cpu_manager.go:414] "containerMap: removing container" podUID="907b0422-9d7d-477f-8d1a-af63f9236529" containerName="revoke-keys" Oct 30 09:13:37.398235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.398003 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907b0422-9d7d-477f-8d1a-af63f9236529" containerName="revoke-keys" Oct 30 09:13:37.398235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.398118 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d8ede7-39da-4cc3-9ef2-302dddde51ea" containerName="revoke-keys" Oct 30 09:13:37.398235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.398129 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="907b0422-9d7d-477f-8d1a-af63f9236529" containerName="revoke-keys" Oct 30 09:13:37.407405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.407375 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.420255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.420191 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:13:37.420396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.420309 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:13:37.420396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.420339 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:13:37.420516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.420406 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:13:37.422650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.422609 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx"] Oct 30 09:13:37.499297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.499255 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.499297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.499300 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.499551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.499475 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jbx\" (UniqueName: \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.514826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.514796 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj"] Oct 30 09:13:37.519203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.519178 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.536621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.536596 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj"] Oct 30 09:13:37.601206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.601206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601157 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8dk\" (UniqueName: \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.601206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601189 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.601522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601339 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.601522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.601522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.601458 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jbx\" (UniqueName: \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.602115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.602096 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.603834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.603811 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.618714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.618688 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jbx\" (UniqueName: \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\") pod \"maas-api-revoke-keys-e2e-ait-70374d4e-llmvx\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.703231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.703181 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.703434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.703260 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8dk\" (UniqueName: \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.703434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.703295 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.704032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.703997 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.706079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.706054 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.721348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.721323 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8dk\" (UniqueName: \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\") pod \"maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:37.737183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.737153 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:37.838671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:37.838395 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:38.038066 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:38.038027 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824965a8_0319_4291_b552_d844342d41dc.slice/crio-945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422 WatchSource:0}: Error finding container 945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422: Status 404 returned error can't find the container with id 945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422 Oct 30 09:13:38.038582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.038548 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx"] Oct 30 09:13:38.155975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.155940 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj"] Oct 30 09:13:38.157486 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:38.157455 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b2292f_4f88_4118_9738_6acecd7c7202.slice/crio-848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9 WatchSource:0}: Error finding container 848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9: Status 404 returned error can't find the container with id 848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9 Oct 30 09:13:38.696953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.696874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:38.806756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.806704 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" event={"ID":"824965a8-0319-4291-b552-d844342d41dc","Type":"ContainerStarted","Data":"7827df439def7acf8046e09c86873986ec51f69d4561446696dab8f874776965"} Oct 30 09:13:38.806946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.806773 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" event={"ID":"824965a8-0319-4291-b552-d844342d41dc","Type":"ContainerStarted","Data":"945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422"} Oct 30 09:13:38.808413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.808382 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" event={"ID":"d2b2292f-4f88-4118-9738-6acecd7c7202","Type":"ContainerStarted","Data":"6d34a677a217e484418e388fa00136ccad8bd9bfb01f69d86e64e9b4c379dc7b"} Oct 30 09:13:38.808523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.808442 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" event={"ID":"d2b2292f-4f88-4118-9738-6acecd7c7202","Type":"ContainerStarted","Data":"848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9"} Oct 30 09:13:38.840312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.840261 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" podStartSLOduration=1.840247752 podStartE2EDuration="1.840247752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:38.838983306 +0000 UTC m=+1182.562410448" watchObservedRunningTime="2026-10-30 09:13:38.840247752 +0000 UTC m=+1182.563674892" Oct 30 09:13:38.874146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:38.874087 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" podStartSLOduration=1.8740688639999998 podStartE2EDuration="1.874068864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:38.87216892 +0000 UTC m=+1182.595596063" watchObservedRunningTime="2026-10-30 09:13:38.874068864 +0000 UTC m=+1182.597496010" Oct 30 09:13:39.012508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.012477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:39.468123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.468082 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:39.501275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.501242 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:39.600454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.600418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:39.702682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.702648 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:39.815899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.815864 4920 generic.go:356] "Generic (PLEG): container finished" podID="d2b2292f-4f88-4118-9738-6acecd7c7202" containerID="6d34a677a217e484418e388fa00136ccad8bd9bfb01f69d86e64e9b4c379dc7b" exitCode=0 Oct 30 09:13:39.816073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.815945 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" event={"ID":"d2b2292f-4f88-4118-9738-6acecd7c7202","Type":"ContainerDied","Data":"6d34a677a217e484418e388fa00136ccad8bd9bfb01f69d86e64e9b4c379dc7b"} Oct 30 09:13:39.817786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.817753 4920 generic.go:356] "Generic (PLEG): container finished" podID="824965a8-0319-4291-b552-d844342d41dc" containerID="7827df439def7acf8046e09c86873986ec51f69d4561446696dab8f874776965" exitCode=0 Oct 30 09:13:39.817900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:39.817833 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" event={"ID":"824965a8-0319-4291-b552-d844342d41dc","Type":"ContainerDied","Data":"7827df439def7acf8046e09c86873986ec51f69d4561446696dab8f874776965"} Oct 30 09:13:40.664596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:40.664566 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:40.994718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:40.994687 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:41.041934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.041903 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:41.047762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.047734 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\") pod \"824965a8-0319-4291-b552-d844342d41dc\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " Oct 30 09:13:41.047893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.047868 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\") pod \"824965a8-0319-4291-b552-d844342d41dc\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " Oct 30 09:13:41.047966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.047926 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\" (UniqueName: \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\") pod \"824965a8-0319-4291-b552-d844342d41dc\" (UID: \"824965a8-0319-4291-b552-d844342d41dc\") " Oct 30 09:13:41.048341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.048309 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca" pod "824965a8-0319-4291-b552-d844342d41dc" (UID: "824965a8-0319-4291-b552-d844342d41dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:41.050110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.050085 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx" pod "824965a8-0319-4291-b552-d844342d41dc" (UID: "824965a8-0319-4291-b552-d844342d41dc"). InnerVolumeSpecName "kube-api-access-t7jbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:41.050184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.050118 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca" pod "824965a8-0319-4291-b552-d844342d41dc" (UID: "824965a8-0319-4291-b552-d844342d41dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:41.148759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.148717 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\") pod \"d2b2292f-4f88-4118-9738-6acecd7c7202\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " Oct 30 09:13:41.148960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.148820 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\" (UniqueName: \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\") pod \"d2b2292f-4f88-4118-9738-6acecd7c7202\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " Oct 30 09:13:41.148960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.148928 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\") pod \"d2b2292f-4f88-4118-9738-6acecd7c7202\" (UID: \"d2b2292f-4f88-4118-9738-6acecd7c7202\") " Oct 30 09:13:41.149090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.149069 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca" pod "d2b2292f-4f88-4118-9738-6acecd7c7202" (UID: "d2b2292f-4f88-4118-9738-6acecd7c7202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:41.149276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.149258 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824965a8-0319-4291-b552-d844342d41dc-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.149276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.149277 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824965a8-0319-4291-b552-d844342d41dc-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.149374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.149287 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jbx\" (UniqueName: \"kubernetes.io/projected/824965a8-0319-4291-b552-d844342d41dc-kube-api-access-t7jbx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.149374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.149296 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.150985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.150965 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca" pod "d2b2292f-4f88-4118-9738-6acecd7c7202" (UID: "d2b2292f-4f88-4118-9738-6acecd7c7202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:41.151048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.151030 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk" pod "d2b2292f-4f88-4118-9738-6acecd7c7202" (UID: "d2b2292f-4f88-4118-9738-6acecd7c7202"). InnerVolumeSpecName "kube-api-access-vj8dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:41.250085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.249996 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b2292f-4f88-4118-9738-6acecd7c7202-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.250085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.250029 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8dk\" (UniqueName: \"kubernetes.io/projected/d2b2292f-4f88-4118-9738-6acecd7c7202-kube-api-access-vj8dk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:41.831686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.831656 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" Oct 30 09:13:41.831886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.831686 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj" event={"ID":"d2b2292f-4f88-4118-9738-6acecd7c7202","Type":"ContainerDied","Data":"848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9"} Oct 30 09:13:41.831886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.831732 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848786a4588d37d3028d2d4caa8dedf1a4c89e78c2b14cfed4b0f1b27b653ad9" Oct 30 09:13:41.833121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.833099 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" event={"ID":"824965a8-0319-4291-b552-d844342d41dc","Type":"ContainerDied","Data":"945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422"} Oct 30 09:13:41.833121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.833104 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx" Oct 30 09:13:41.833292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:41.833133 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945137ea8fdbda882708c61d9b13324387e81bcd79cfb67cddf9390000ee1422" Oct 30 09:13:42.297385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.297321 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" podUID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" containerName="payload-processing" containerID="cri-o://b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10" gracePeriod=25 Oct 30 09:13:42.332485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.332448 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" podUID="e688bbf5-b10e-49aa-9f4c-462467e013ba" containerName="payload-pre-processing" containerID="cri-o://e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab" gracePeriod=25 Oct 30 09:13:42.409007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.408958 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:42.609824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.609797 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:42.625623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.625597 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:42.665695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.665667 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\") pod \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " Oct 30 09:13:42.665896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.665716 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\" (UniqueName: \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\") pod \"e688bbf5-b10e-49aa-9f4c-462467e013ba\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " Oct 30 09:13:42.665896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.665756 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\") pod \"e688bbf5-b10e-49aa-9f4c-462467e013ba\" (UID: \"e688bbf5-b10e-49aa-9f4c-462467e013ba\") " Oct 30 09:13:42.665896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.665831 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\" (UniqueName: \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\") pod \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\" (UID: \"5c2cf9ac-abe6-4dd0-8c51-a83b63917284\") " Oct 30 09:13:42.666181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.666153 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume" pod "5c2cf9ac-abe6-4dd0-8c51-a83b63917284" (UID: "5c2cf9ac-abe6-4dd0-8c51-a83b63917284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:42.666325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.666286 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume" pod "e688bbf5-b10e-49aa-9f4c-462467e013ba" (UID: "e688bbf5-b10e-49aa-9f4c-462467e013ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:42.668070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.668046 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c" pod "e688bbf5-b10e-49aa-9f4c-462467e013ba" (UID: "e688bbf5-b10e-49aa-9f4c-462467e013ba"). InnerVolumeSpecName "kube-api-access-pwh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:42.668632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.668614 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k" pod "5c2cf9ac-abe6-4dd0-8c51-a83b63917284" (UID: "5c2cf9ac-abe6-4dd0-8c51-a83b63917284"). InnerVolumeSpecName "kube-api-access-nk74k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:42.766898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.766860 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk74k\" (UniqueName: \"kubernetes.io/projected/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-kube-api-access-nk74k\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:42.766898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.766892 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2cf9ac-abe6-4dd0-8c51-a83b63917284-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:42.766898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.766902 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwh8c\" (UniqueName: \"kubernetes.io/projected/e688bbf5-b10e-49aa-9f4c-462467e013ba-kube-api-access-pwh8c\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:42.767144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.766911 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e688bbf5-b10e-49aa-9f4c-462467e013ba-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:42.841491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.841404 4920 generic.go:356] "Generic (PLEG): container finished" podID="e688bbf5-b10e-49aa-9f4c-462467e013ba" containerID="e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab" exitCode=0 Oct 30 09:13:42.841491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.841451 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" Oct 30 09:13:42.841734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.841486 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" event={"ID":"e688bbf5-b10e-49aa-9f4c-462467e013ba","Type":"ContainerDied","Data":"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab"} Oct 30 09:13:42.841734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.841557 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj" event={"ID":"e688bbf5-b10e-49aa-9f4c-462467e013ba","Type":"ContainerDied","Data":"8aa4af75558e240ce4d972ee9fd84a410e07c93322417c9d846802874b2dbb7c"} Oct 30 09:13:42.841734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.841589 4920 scope.go:122] "RemoveContainer" containerID="e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab" Oct 30 09:13:42.843050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.843027 4920 generic.go:356] "Generic (PLEG): container finished" podID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" containerID="b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10" exitCode=0 Oct 30 09:13:42.843174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.843064 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" event={"ID":"5c2cf9ac-abe6-4dd0-8c51-a83b63917284","Type":"ContainerDied","Data":"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10"} Oct 30 09:13:42.843174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.843094 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" event={"ID":"5c2cf9ac-abe6-4dd0-8c51-a83b63917284","Type":"ContainerDied","Data":"94e77e37a38862e92a705d372274b87270b937466ff69302bf994b9ad6227d67"} Oct 30 09:13:42.843174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.843067 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf" Oct 30 09:13:42.852066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.852048 4920 scope.go:122] "RemoveContainer" containerID="e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab" Oct 30 09:13:42.852434 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:42.852402 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab\": container with ID starting with e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab not found: ID does not exist" containerID="e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab" Oct 30 09:13:42.852528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.852435 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab"} err="failed to get container status \"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab\": rpc error: code = NotFound desc = could not find container \"e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab\": container with ID starting with e8e641b4bcd494b43b3e116940b22956848363d4347a35cc69b5b67842bb36ab not found: ID does not exist" Oct 30 09:13:42.852528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.852454 4920 scope.go:122] "RemoveContainer" containerID="b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10" Oct 30 09:13:42.862743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.862660 4920 scope.go:122] "RemoveContainer" containerID="b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10" Oct 30 09:13:42.863091 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:42.863022 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10\": container with ID starting with b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10 not found: ID does not exist" containerID="b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10" Oct 30 09:13:42.863091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.863056 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10"} err="failed to get container status \"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10\": rpc error: code = NotFound desc = could not find container \"b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10\": container with ID starting with b21c87cec19b8ba239a0a10ebdfc51e41674f71b36e3473e9ec712894567ee10 not found: ID does not exist" Oct 30 09:13:42.886195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.886161 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:42.888886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.888857 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ffc2a58-9857b554d-qffqf"] Oct 30 09:13:42.916614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.916512 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:42.919896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:42.919869 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ffc2a58-7656899dcf-cs2pj"] Oct 30 09:13:44.172422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.172382 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp"] Oct 30 09:13:44.173129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173109 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e688bbf5-b10e-49aa-9f4c-462467e013ba" containerName="payload-pre-processing" Oct 30 09:13:44.173187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173132 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e688bbf5-b10e-49aa-9f4c-462467e013ba" containerName="payload-pre-processing" Oct 30 09:13:44.173187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173147 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d2b2292f-4f88-4118-9738-6acecd7c7202" containerName="revoke-keys" Oct 30 09:13:44.173187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173154 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b2292f-4f88-4118-9738-6acecd7c7202" containerName="revoke-keys" Oct 30 09:13:44.173187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173176 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" containerName="payload-processing" Oct 30 09:13:44.173187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173183 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" containerName="payload-processing" Oct 30 09:13:44.173382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173225 4920 cpu_manager.go:414] "containerMap: removing container" podUID="824965a8-0319-4291-b552-d844342d41dc" containerName="revoke-keys" Oct 30 09:13:44.173382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173233 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824965a8-0319-4291-b552-d844342d41dc" containerName="revoke-keys" Oct 30 09:13:44.173382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173348 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="824965a8-0319-4291-b552-d844342d41dc" containerName="revoke-keys" Oct 30 09:13:44.173382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173366 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e688bbf5-b10e-49aa-9f4c-462467e013ba" containerName="payload-pre-processing" Oct 30 09:13:44.173382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173374 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b2292f-4f88-4118-9738-6acecd7c7202" containerName="revoke-keys" Oct 30 09:13:44.173526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.173384 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" containerName="payload-processing" Oct 30 09:13:44.178428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.178410 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.193294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.193265 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp"] Oct 30 09:13:44.203383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.203349 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:13:44.204417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.204395 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:13:44.204541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.204527 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:13:44.204866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.204848 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:13:44.281798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.281760 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.282007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.281814 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.282007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.281935 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6k6\" (UniqueName: \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.382486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.382453 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.382665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.382496 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.382665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.382552 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6k6\" (UniqueName: \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.383095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.383067 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.384866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.384838 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.398522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.398491 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2cf9ac-abe6-4dd0-8c51-a83b63917284" path="/var/lib/kubelet/pods/5c2cf9ac-abe6-4dd0-8c51-a83b63917284/volumes" Oct 30 09:13:44.399043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.399019 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e688bbf5-b10e-49aa-9f4c-462467e013ba" path="/var/lib/kubelet/pods/e688bbf5-b10e-49aa-9f4c-462467e013ba/volumes" Oct 30 09:13:44.399504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.399485 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6k6\" (UniqueName: \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\") pod \"maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.469251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.469153 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" podUID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" containerName="payload-processing" containerID="cri-o://3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e" gracePeriod=25 Oct 30 09:13:44.492255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.492229 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:44.501954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.501927 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" podUID="a066df90-b698-483b-9b74-3f0f0f6fa43a" containerName="payload-pre-processing" containerID="cri-o://305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291" gracePeriod=25 Oct 30 09:13:44.526505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.526464 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:44.756068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.756041 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:44.778691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.778666 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:44.786747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.786724 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\") pod \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " Oct 30 09:13:44.786888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.786829 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\" (UniqueName: \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\") pod \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\" (UID: \"0fa8f70d-9cda-4abb-951c-ba6efb988a3e\") " Oct 30 09:13:44.787168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.787146 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume" pod "0fa8f70d-9cda-4abb-951c-ba6efb988a3e" (UID: "0fa8f70d-9cda-4abb-951c-ba6efb988a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:44.787316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.787295 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:44.787983 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:44.787963 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb907a46e_4d03_419c_a734_bb215d3504be.slice/crio-92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61 WatchSource:0}: Error finding container 92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61: Status 404 returned error can't find the container with id 92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61 Oct 30 09:13:44.788717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.788701 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp"] Oct 30 09:13:44.788963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.788943 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn" pod "0fa8f70d-9cda-4abb-951c-ba6efb988a3e" (UID: "0fa8f70d-9cda-4abb-951c-ba6efb988a3e"). InnerVolumeSpecName "kube-api-access-zctvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:44.860748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.860716 4920 generic.go:356] "Generic (PLEG): container finished" podID="a066df90-b698-483b-9b74-3f0f0f6fa43a" containerID="305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291" exitCode=0 Oct 30 09:13:44.860907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.860757 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" Oct 30 09:13:44.860907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.860791 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" event={"ID":"a066df90-b698-483b-9b74-3f0f0f6fa43a","Type":"ContainerDied","Data":"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291"} Oct 30 09:13:44.860907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.860843 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46" event={"ID":"a066df90-b698-483b-9b74-3f0f0f6fa43a","Type":"ContainerDied","Data":"7388c110cc362fcb561892eee7f2ce9d6206e69ecf0433b2664d29bb15949e71"} Oct 30 09:13:44.860907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.860873 4920 scope.go:122] "RemoveContainer" containerID="305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291" Oct 30 09:13:44.862133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.862112 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" event={"ID":"b907a46e-4d03-419c-a734-bb215d3504be","Type":"ContainerStarted","Data":"92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61"} Oct 30 09:13:44.863455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.863429 4920 generic.go:356] "Generic (PLEG): container finished" podID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" containerID="3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e" exitCode=0 Oct 30 09:13:44.863571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.863478 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" Oct 30 09:13:44.863571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.863509 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" event={"ID":"0fa8f70d-9cda-4abb-951c-ba6efb988a3e","Type":"ContainerDied","Data":"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e"} Oct 30 09:13:44.863571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.863559 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn" event={"ID":"0fa8f70d-9cda-4abb-951c-ba6efb988a3e","Type":"ContainerDied","Data":"743904545ea88dccb0931f3fd6ce9dd89750ce4489e41a743a1e8d5cec934f9b"} Oct 30 09:13:44.871489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.871465 4920 scope.go:122] "RemoveContainer" containerID="305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291" Oct 30 09:13:44.871854 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:44.871834 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291\": container with ID starting with 305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291 not found: ID does not exist" containerID="305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291" Oct 30 09:13:44.871941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.871862 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291"} err="failed to get container status \"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291\": rpc error: code = NotFound desc = could not find container \"305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291\": container with ID starting with 305469d831e458432d46298376887fd3c24ae71ba75b2b257dd91338c9a47291 not found: ID does not exist" Oct 30 09:13:44.871941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.871879 4920 scope.go:122] "RemoveContainer" containerID="3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e" Oct 30 09:13:44.881818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.881205 4920 scope.go:122] "RemoveContainer" containerID="3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e" Oct 30 09:13:44.882092 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:44.882074 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e\": container with ID starting with 3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e not found: ID does not exist" containerID="3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e" Oct 30 09:13:44.882140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.882101 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e"} err="failed to get container status \"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e\": rpc error: code = NotFound desc = could not find container \"3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e\": container with ID starting with 3145440eac4160277bda517e3abd15ae08b54279cc57663fa57f1405378f825e not found: ID does not exist" Oct 30 09:13:44.887874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.887857 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\" (UniqueName: \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\") pod \"a066df90-b698-483b-9b74-3f0f0f6fa43a\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " Oct 30 09:13:44.887970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.887926 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\") pod \"a066df90-b698-483b-9b74-3f0f0f6fa43a\" (UID: \"a066df90-b698-483b-9b74-3f0f0f6fa43a\") " Oct 30 09:13:44.888287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.888264 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume" pod "a066df90-b698-483b-9b74-3f0f0f6fa43a" (UID: "a066df90-b698-483b-9b74-3f0f0f6fa43a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:44.888377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.888330 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a066df90-b698-483b-9b74-3f0f0f6fa43a-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:44.888377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.888346 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zctvn\" (UniqueName: \"kubernetes.io/projected/0fa8f70d-9cda-4abb-951c-ba6efb988a3e-kube-api-access-zctvn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:44.889849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.889833 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs" pod "a066df90-b698-483b-9b74-3f0f0f6fa43a" (UID: "a066df90-b698-483b-9b74-3f0f0f6fa43a"). InnerVolumeSpecName "kube-api-access-587qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:44.903939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.903915 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:44.907251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.907209 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c806f9c-7b89fbdffb-vsxsn"] Oct 30 09:13:44.989749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:44.989672 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-587qs\" (UniqueName: \"kubernetes.io/projected/a066df90-b698-483b-9b74-3f0f0f6fa43a-kube-api-access-587qs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:45.204253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:45.204137 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:45.207382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:45.207357 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c806f9c-57c6d67b55-nkz46"] Oct 30 09:13:45.870947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:45.870911 4920 generic.go:356] "Generic (PLEG): container finished" podID="b907a46e-4d03-419c-a734-bb215d3504be" containerID="3cb5f9623f7180b83f02395190e436e7ecf0c9d5f00d1c97776f638a4fa0c27d" exitCode=0 Oct 30 09:13:45.871147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:45.870951 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" event={"ID":"b907a46e-4d03-419c-a734-bb215d3504be","Type":"ContainerDied","Data":"3cb5f9623f7180b83f02395190e436e7ecf0c9d5f00d1c97776f638a4fa0c27d"} Oct 30 09:13:46.128598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.128524 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:46.401072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.400997 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" path="/var/lib/kubelet/pods/0fa8f70d-9cda-4abb-951c-ba6efb988a3e/volumes" Oct 30 09:13:46.401452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.401393 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a066df90-b698-483b-9b74-3f0f0f6fa43a" path="/var/lib/kubelet/pods/a066df90-b698-483b-9b74-3f0f0f6fa43a/volumes" Oct 30 09:13:46.554020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.552105 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:46.704872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.704772 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:46.802503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.802462 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:46.889262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.889232 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw"] Oct 30 09:13:46.899498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:46.899467 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ffc2a58-q57lw"] Oct 30 09:13:47.013255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.011104 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:47.013255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.011349 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" containerID="cri-o://de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36" gracePeriod=30 Oct 30 09:13:47.046809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.046779 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:47.060427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.060400 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:47.111102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.111062 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\") pod \"b907a46e-4d03-419c-a734-bb215d3504be\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " Oct 30 09:13:47.111326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.111292 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\" (UniqueName: \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\") pod \"b907a46e-4d03-419c-a734-bb215d3504be\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " Oct 30 09:13:47.111411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.111334 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\") pod \"b907a46e-4d03-419c-a734-bb215d3504be\" (UID: \"b907a46e-4d03-419c-a734-bb215d3504be\") " Oct 30 09:13:47.111508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.111479 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca" pod "b907a46e-4d03-419c-a734-bb215d3504be" (UID: "b907a46e-4d03-419c-a734-bb215d3504be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:47.111705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.111685 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.113544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.113515 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca" pod "b907a46e-4d03-419c-a734-bb215d3504be" (UID: "b907a46e-4d03-419c-a734-bb215d3504be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:47.113544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.113520 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6" pod "b907a46e-4d03-419c-a734-bb215d3504be" (UID: "b907a46e-4d03-419c-a734-bb215d3504be"). InnerVolumeSpecName "kube-api-access-rq6k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:47.212980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.212889 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq6k6\" (UniqueName: \"kubernetes.io/projected/b907a46e-4d03-419c-a734-bb215d3504be-kube-api-access-rq6k6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.212980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.212919 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907a46e-4d03-419c-a734-bb215d3504be-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.510944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.510862 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:47.701486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.701451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:47.808183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.808140 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t"] Oct 30 09:13:47.817656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.817624 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c806f9c-pvt2t"] Oct 30 09:13:47.887559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.887532 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" Oct 30 09:13:47.887559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.887552 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:47.887769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.887530 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp" event={"ID":"b907a46e-4d03-419c-a734-bb215d3504be","Type":"ContainerDied","Data":"92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61"} Oct 30 09:13:47.887769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.887660 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92012fc689533c9f9a141266aeab1d8bec4f00fe9ae432296d2aa5ef29684b61" Oct 30 09:13:47.985859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:47.985827 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:48.382337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.382297 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:13:48.383088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383061 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" containerName="payload-processing" Oct 30 09:13:48.383088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383088 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" containerName="payload-processing" Oct 30 09:13:48.383292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383123 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a066df90-b698-483b-9b74-3f0f0f6fa43a" containerName="payload-pre-processing" Oct 30 09:13:48.383292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383132 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a066df90-b698-483b-9b74-3f0f0f6fa43a" containerName="payload-pre-processing" Oct 30 09:13:48.383292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383158 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b907a46e-4d03-419c-a734-bb215d3504be" containerName="revoke-keys" Oct 30 09:13:48.383292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383166 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b907a46e-4d03-419c-a734-bb215d3504be" containerName="revoke-keys" Oct 30 09:13:48.383502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383302 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa8f70d-9cda-4abb-951c-ba6efb988a3e" containerName="payload-processing" Oct 30 09:13:48.383502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383319 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a066df90-b698-483b-9b74-3f0f0f6fa43a" containerName="payload-pre-processing" Oct 30 09:13:48.383502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.383332 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b907a46e-4d03-419c-a734-bb215d3504be" containerName="revoke-keys" Oct 30 09:13:48.387277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.387252 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.417374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.417337 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e3d2ff-openshift-default-dockercfg-j9w6q\"" Oct 30 09:13:48.417688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.417660 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d8ede7-39da-4cc3-9ef2-302dddde51ea" path="/var/lib/kubelet/pods/29d8ede7-39da-4cc3-9ef2-302dddde51ea/volumes" Oct 30 09:13:48.418336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.418313 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907b0422-9d7d-477f-8d1a-af63f9236529" path="/var/lib/kubelet/pods/907b0422-9d7d-477f-8d1a-af63f9236529/volumes" Oct 30 09:13:48.419339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.419316 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:13:48.427968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.427939 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvrw\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.427991 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428075 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428166 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428201 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428268 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428304 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428378 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.428724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.428413 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.445022 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.444992 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:48.445166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.445088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:48.467708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.467680 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:48.529959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.529919 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.529959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.529968 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.529998 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530038 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530075 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530578 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530634 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530741 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530783 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.530993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.530860 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvrw\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.531115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.531061 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.531441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.531419 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.531814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.531791 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.532294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.532274 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.533027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.533005 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.534460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.534392 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.547612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.547582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:48.549093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.549070 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.549275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.549253 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvrw\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:48.645677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.645592 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:48.705559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:48.705519 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:49.026941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.026913 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:13:49.028955 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:49.028921 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e00b46_5f10_4079_87a8_8e063fba6cbc.slice/crio-17e7c33752bbd7b5e2b92313bdd3b28e48085105c7037846889dea0caa0bb1fc WatchSource:0}: Error finding container 17e7c33752bbd7b5e2b92313bdd3b28e48085105c7037846889dea0caa0bb1fc: Status 404 returned error can't find the container with id 17e7c33752bbd7b5e2b92313bdd3b28e48085105c7037846889dea0caa0bb1fc Oct 30 09:13:49.031615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.031577 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:49.031696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.031666 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:49.031732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.031706 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:49.275905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.275806 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:49.291083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.291051 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.317375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.317341 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:49.414878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.414838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.511497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.511463 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.599779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.599736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.739887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.739854 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:13:49.744720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.744692 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.759594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.759569 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f69c8-openshift-default-dockercfg-hmrnb\"" Oct 30 09:13:49.774956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.774926 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:13:49.803289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.803249 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.845834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.845802 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:13:49.846348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846313 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846355 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846384 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4577t\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846525 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846553 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846659 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846684 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.846849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.846789 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.902547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.902355 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerStarted","Data":"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8"} Oct 30 09:13:49.902547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.902423 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerStarted","Data":"17e7c33752bbd7b5e2b92313bdd3b28e48085105c7037846889dea0caa0bb1fc"} Oct 30 09:13:49.908666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.908631 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947597 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947638 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947664 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947746 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947783 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947816 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947856 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4577t\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.947900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947883 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.948470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.947931 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.948470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.948288 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.948577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.948494 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.948727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.948702 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.949254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.949200 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.950617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.950576 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.953384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.953359 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.955241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.955202 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.981543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.981499 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4577t\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:49.983262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:49.983233 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:50.060771 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.060737 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:50.240590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.240505 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:50.392816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.392753 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podStartSLOduration=2.392732292 podStartE2EDuration="2.392732292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:49.967105977 +0000 UTC m=+1193.690533141" watchObservedRunningTime="2026-10-30 09:13:50.392732292 +0000 UTC m=+1194.116159437" Oct 30 09:13:50.396140 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:50.396110 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6ed2a7_3d04_461c_a1bd_f1e2068b2ba2.slice/crio-4042920146ab69825fe47ee0e36ada792cbe6e7e490d4d8a61867e6b15f9eb6e WatchSource:0}: Error finding container 4042920146ab69825fe47ee0e36ada792cbe6e7e490d4d8a61867e6b15f9eb6e: Status 404 returned error can't find the container with id 4042920146ab69825fe47ee0e36ada792cbe6e7e490d4d8a61867e6b15f9eb6e Oct 30 09:13:50.405106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.405062 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:50.405394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.405370 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:50.405559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.405537 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:50.409467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.409443 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:13:50.706732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.706702 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:50.713019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.712989 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:50.760959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.760929 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:50.873237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.873186 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx"] Oct 30 09:13:50.877286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.877259 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-70374d4e-llmvx"] Oct 30 09:13:50.912826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.912789 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerStarted","Data":"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3"} Oct 30 09:13:50.913036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.912868 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerStarted","Data":"4042920146ab69825fe47ee0e36ada792cbe6e7e490d4d8a61867e6b15f9eb6e"} Oct 30 09:13:50.913135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.913117 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:50.914103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.914080 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:13:50.916993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.916966 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:50.958360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:50.958245 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podStartSLOduration=1.9582094300000001 podStartE2EDuration="1.95820943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:50.955798632 +0000 UTC m=+1194.679225777" watchObservedRunningTime="2026-10-30 09:13:50.95820943 +0000 UTC m=+1194.681636578" Oct 30 09:13:51.060907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.060864 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:51.067656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.067622 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:51.532333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.532295 4920 patch_prober.go:29] interesting pod/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:13:51.532487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.532359 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:13:51.681035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.680998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:51.920792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.920757 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:51.921814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:51.921789 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:13:52.236937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.236899 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.281885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.281851 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:52.329872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.329837 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.367495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.367457 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj"] Oct 30 09:13:52.370451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.370420 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6233a0dd-9qtrj"] Oct 30 09:13:52.371688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371667 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.371790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371718 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.371790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371758 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.371903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371787 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.371903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371841 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.371903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371881 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.372053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.371960 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.372153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372123 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:52.372153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372137 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:52.372387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372143 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.372387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372253 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\") pod \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\" (UID: \"8596595e-3cb5-4dce-aa66-0a507f1e6c23\") " Oct 30 09:13:52.372387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372260 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:52.372556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372397 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:52.372620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372595 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:52.372790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372768 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.372858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372796 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.372858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372813 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.372858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372826 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.372858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.372840 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.374128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.374102 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "kube-api-access-wksfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:52.374402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.374374 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:52.374565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.374542 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:52.374632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.374614 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo" pod "8596595e-3cb5-4dce-aa66-0a507f1e6c23" (UID: "8596595e-3cb5-4dce-aa66-0a507f1e6c23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:13:52.399175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.399141 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824965a8-0319-4291-b552-d844342d41dc" path="/var/lib/kubelet/pods/824965a8-0319-4291-b552-d844342d41dc/volumes" Oct 30 09:13:52.399601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.399586 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b2292f-4f88-4118-9738-6acecd7c7202" path="/var/lib/kubelet/pods/d2b2292f-4f88-4118-9738-6acecd7c7202/volumes" Oct 30 09:13:52.411955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.411922 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.475252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.474453 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.475252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.474559 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wksfb\" (UniqueName: \"kubernetes.io/projected/8596595e-3cb5-4dce-aa66-0a507f1e6c23-kube-api-access-wksfb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.475252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.474574 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.475252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.474587 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596595e-3cb5-4dce-aa66-0a507f1e6c23-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.523707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.523632 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.630097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.630062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.755292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.752068 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:52.929655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.929622 4920 generic.go:356] "Generic (PLEG): container finished" podID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerID="de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36" exitCode=0 Oct 30 09:13:52.930079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.929677 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" Oct 30 09:13:52.930079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.929712 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerDied","Data":"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36"} Oct 30 09:13:52.930079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.929771 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq" event={"ID":"8596595e-3cb5-4dce-aa66-0a507f1e6c23","Type":"ContainerDied","Data":"48658854ec79ac5ed5a763ef30dc7b14c827e00bf92c1a45d0a7e7344ef2e8e9"} Oct 30 09:13:52.930079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.929801 4920 scope.go:122] "RemoveContainer" containerID="de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36" Oct 30 09:13:52.948810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.948781 4920 scope.go:122] "RemoveContainer" containerID="149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059" Oct 30 09:13:52.958000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.957979 4920 scope.go:122] "RemoveContainer" containerID="de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36" Oct 30 09:13:52.958312 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:52.958287 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36\": container with ID starting with de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36 not found: ID does not exist" containerID="de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36" Oct 30 09:13:52.958398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.958319 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36"} err="failed to get container status \"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36\": rpc error: code = NotFound desc = could not find container \"de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36\": container with ID starting with de3a3d569661a5ba163b7ed11bce3dc24cb01c7026cb00bbcab73fba88c7af36 not found: ID does not exist" Oct 30 09:13:52.958398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.958337 4920 scope.go:122] "RemoveContainer" containerID="149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059" Oct 30 09:13:52.958565 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:52.958547 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059\": container with ID starting with 149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059 not found: ID does not exist" containerID="149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059" Oct 30 09:13:52.958604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.958571 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059"} err="failed to get container status \"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059\": rpc error: code = NotFound desc = could not find container \"149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059\": container with ID starting with 149a7fc41c3280e3eb6c50c9856e439fdaf3f673f0671937298369116c197059 not found: ID does not exist" Oct 30 09:13:52.974353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.974327 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:52.984352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:52.984326 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ffc2a58-openshift-default-6b69db958c-n2bpq"] Oct 30 09:13:53.317840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.317797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:53.418796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.418746 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:53.445552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.445469 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" podUID="08c1c128-233a-4edb-913f-cf97b4d138b9" containerName="payload-processing" containerID="cri-o://7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf" gracePeriod=25 Oct 30 09:13:53.468304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.468255 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" podUID="239411df-a827-4427-bd9b-9210b81781b9" containerName="payload-pre-processing" containerID="cri-o://5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9" gracePeriod=25 Oct 30 09:13:53.566705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.566668 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:53.675964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.675934 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:53.762038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.762014 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:53.764944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.764925 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:53.885636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.885546 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:13:53.886098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886080 4920 cpu_manager.go:414] "containerMap: removing container" podUID="08c1c128-233a-4edb-913f-cf97b4d138b9" containerName="payload-processing" Oct 30 09:13:53.886192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886102 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c1c128-233a-4edb-913f-cf97b4d138b9" containerName="payload-processing" Oct 30 09:13:53.886192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886121 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:53.886192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886128 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:53.886192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886140 4920 cpu_manager.go:414] "containerMap: removing container" podUID="239411df-a827-4427-bd9b-9210b81781b9" containerName="payload-pre-processing" Oct 30 09:13:53.886192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886147 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239411df-a827-4427-bd9b-9210b81781b9" containerName="payload-pre-processing" Oct 30 09:13:53.886451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886325 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:53.886451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886345 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c1c128-233a-4edb-913f-cf97b4d138b9" containerName="payload-processing" Oct 30 09:13:53.886451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886355 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:53.886451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.886364 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="239411df-a827-4427-bd9b-9210b81781b9" containerName="payload-pre-processing" Oct 30 09:13:53.889775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.889746 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\") pod \"08c1c128-233a-4edb-913f-cf97b4d138b9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " Oct 30 09:13:53.889927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.889782 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\" (UniqueName: \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\") pod \"08c1c128-233a-4edb-913f-cf97b4d138b9\" (UID: \"08c1c128-233a-4edb-913f-cf97b4d138b9\") " Oct 30 09:13:53.889927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.889822 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\") pod \"239411df-a827-4427-bd9b-9210b81781b9\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " Oct 30 09:13:53.889927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.889892 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\" (UniqueName: \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\") pod \"239411df-a827-4427-bd9b-9210b81781b9\" (UID: \"239411df-a827-4427-bd9b-9210b81781b9\") " Oct 30 09:13:53.890185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.890167 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.890581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.890557 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume" pod "239411df-a827-4427-bd9b-9210b81781b9" (UID: "239411df-a827-4427-bd9b-9210b81781b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:53.890943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.890917 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume" pod "08c1c128-233a-4edb-913f-cf97b4d138b9" (UID: "08c1c128-233a-4edb-913f-cf97b4d138b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:53.892637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.892610 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m" pod "08c1c128-233a-4edb-913f-cf97b4d138b9" (UID: "08c1c128-233a-4edb-913f-cf97b4d138b9"). InnerVolumeSpecName "kube-api-access-m7s6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:53.892801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.892747 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9" pod "239411df-a827-4427-bd9b-9210b81781b9" (UID: "239411df-a827-4427-bd9b-9210b81781b9"). InnerVolumeSpecName "kube-api-access-qx2c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:53.903240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.903188 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3a61e1-openshift-default-dockercfg-vvdt5\"" Oct 30 09:13:53.915027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.914997 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:13:53.938201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.938164 4920 generic.go:356] "Generic (PLEG): container finished" podID="239411df-a827-4427-bd9b-9210b81781b9" containerID="5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9" exitCode=0 Oct 30 09:13:53.938692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.938249 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" Oct 30 09:13:53.938692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.938342 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" event={"ID":"239411df-a827-4427-bd9b-9210b81781b9","Type":"ContainerDied","Data":"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9"} Oct 30 09:13:53.938692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.938397 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll" event={"ID":"239411df-a827-4427-bd9b-9210b81781b9","Type":"ContainerDied","Data":"4b6d74821c2b4d1bdf6f63489ff4d03fb16871fb62d04b1d1e7f0172506509e6"} Oct 30 09:13:53.938692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.938429 4920 scope.go:122] "RemoveContainer" containerID="5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9" Oct 30 09:13:53.941564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.941537 4920 generic.go:356] "Generic (PLEG): container finished" podID="08c1c128-233a-4edb-913f-cf97b4d138b9" containerID="7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf" exitCode=0 Oct 30 09:13:53.941695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.941584 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" Oct 30 09:13:53.941695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.941613 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" event={"ID":"08c1c128-233a-4edb-913f-cf97b4d138b9","Type":"ContainerDied","Data":"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf"} Oct 30 09:13:53.941695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.941665 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9" event={"ID":"08c1c128-233a-4edb-913f-cf97b4d138b9","Type":"ContainerDied","Data":"c0b6c498448e9324d1dec50cb5e9fd3eea5e31210dfe0e9ef6555da1d4aa03a2"} Oct 30 09:13:53.945655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.945628 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:53.950438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.950410 4920 scope.go:122] "RemoveContainer" containerID="5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9" Oct 30 09:13:53.950892 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:53.950866 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9\": container with ID starting with 5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9 not found: ID does not exist" containerID="5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9" Oct 30 09:13:53.950994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.950900 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9"} err="failed to get container status \"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9\": rpc error: code = NotFound desc = could not find container \"5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9\": container with ID starting with 5a5ca213619316b796c3491173e0b6426d5eee6282b75cb0339a8c5a2a0936f9 not found: ID does not exist" Oct 30 09:13:53.950994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.950924 4920 scope.go:122] "RemoveContainer" containerID="7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf" Oct 30 09:13:53.961976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.961961 4920 scope.go:122] "RemoveContainer" containerID="7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf" Oct 30 09:13:53.962272 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:53.962253 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf\": container with ID starting with 7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf not found: ID does not exist" containerID="7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf" Oct 30 09:13:53.962331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.962280 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf"} err="failed to get container status \"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf\": rpc error: code = NotFound desc = could not find container \"7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf\": container with ID starting with 7edf3816522913894ca390326b0876c6ed747d456fd46347572ef33df13173bf not found: ID does not exist" Oct 30 09:13:53.990998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.990952 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991071 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991134 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzk27\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991293 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991351 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991373 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991446 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991535 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991555 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:53.991677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991633 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/239411df-a827-4427-bd9b-9210b81781b9-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.991677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991644 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2c9\" (UniqueName: \"kubernetes.io/projected/239411df-a827-4427-bd9b-9210b81781b9-kube-api-access-qx2c9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.991677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991661 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c1c128-233a-4edb-913f-cf97b4d138b9-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.991677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.991673 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7s6m\" (UniqueName: \"kubernetes.io/projected/08c1c128-233a-4edb-913f-cf97b4d138b9-kube-api-access-m7s6m\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.994034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.994012 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:53.998074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:53.998048 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6233a0dd-8c6f95df4-jjpk9"] Oct 30 09:13:54.037249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.037197 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:54.040546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.040517 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6233a0dd-55479bf67-s2rll"] Oct 30 09:13:54.086971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.086924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.092295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092263 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092320 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092349 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzk27\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092379 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092405 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092434 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092462 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092520 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092550 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.092766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.092726 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.093300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.093017 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.093300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.093138 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.093469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.093322 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.093529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.093466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.095096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.095075 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.095377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.095359 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.112568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.112540 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.113426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.113408 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzk27\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.183932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.183821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.221327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.221290 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:54.277253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.276320 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" podUID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" containerName="payload-processing" containerID="cri-o://4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3" gracePeriod=25 Oct 30 09:13:54.293558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.291835 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.330715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.330483 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" podUID="15fd7d94-a7c2-4855-8256-365af8b51408" containerName="payload-pre-processing" containerID="cri-o://1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f" gracePeriod=25 Oct 30 09:13:54.389305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.389273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.400135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.400103 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c1c128-233a-4edb-913f-cf97b4d138b9" path="/var/lib/kubelet/pods/08c1c128-233a-4edb-913f-cf97b4d138b9/volumes" Oct 30 09:13:54.400563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.400543 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239411df-a827-4427-bd9b-9210b81781b9" path="/var/lib/kubelet/pods/239411df-a827-4427-bd9b-9210b81781b9/volumes" Oct 30 09:13:54.400994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.400975 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" path="/var/lib/kubelet/pods/8596595e-3cb5-4dce-aa66-0a507f1e6c23/volumes" Oct 30 09:13:54.490339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.490302 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.585582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.585546 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:13:54.587057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.586970 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:54.587057 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:54.587026 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89df4fc_4e83_421d_881f_56e808d9f1f5.slice/crio-defbdeecab7e3c493067e998f2393ed13a6e6c5b791e936a8758423510404e6b WatchSource:0}: Error finding container defbdeecab7e3c493067e998f2393ed13a6e6c5b791e936a8758423510404e6b: Status 404 returned error can't find the container with id defbdeecab7e3c493067e998f2393ed13a6e6c5b791e936a8758423510404e6b Oct 30 09:13:54.590152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.590113 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.590278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.590200 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.590278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.590264 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.616536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.616507 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:54.649582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.649554 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:54.725318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.723640 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\") pod \"15fd7d94-a7c2-4855-8256-365af8b51408\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " Oct 30 09:13:54.725318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.723724 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\" (UniqueName: \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\") pod \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " Oct 30 09:13:54.725318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.723786 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\") pod \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\" (UID: \"d63a8edd-828b-4696-b88e-f0a384ec1fb5\") " Oct 30 09:13:54.725318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.723909 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\" (UniqueName: \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\") pod \"15fd7d94-a7c2-4855-8256-365af8b51408\" (UID: \"15fd7d94-a7c2-4855-8256-365af8b51408\") " Oct 30 09:13:54.725670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.725452 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume" pod "15fd7d94-a7c2-4855-8256-365af8b51408" (UID: "15fd7d94-a7c2-4855-8256-365af8b51408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:54.725670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.725623 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume" pod "d63a8edd-828b-4696-b88e-f0a384ec1fb5" (UID: "d63a8edd-828b-4696-b88e-f0a384ec1fb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:54.734596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.734532 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh" pod "15fd7d94-a7c2-4855-8256-365af8b51408" (UID: "15fd7d94-a7c2-4855-8256-365af8b51408"). InnerVolumeSpecName "kube-api-access-dxfzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:54.734970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.734936 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6" pod "d63a8edd-828b-4696-b88e-f0a384ec1fb5" (UID: "d63a8edd-828b-4696-b88e-f0a384ec1fb5"). InnerVolumeSpecName "kube-api-access-gt9l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:54.825804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.825768 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxfzh\" (UniqueName: \"kubernetes.io/projected/15fd7d94-a7c2-4855-8256-365af8b51408-kube-api-access-dxfzh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:54.825804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.825803 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fd7d94-a7c2-4855-8256-365af8b51408-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:54.825804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.825821 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9l6\" (UniqueName: \"kubernetes.io/projected/d63a8edd-828b-4696-b88e-f0a384ec1fb5-kube-api-access-gt9l6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:54.826050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.825833 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d63a8edd-828b-4696-b88e-f0a384ec1fb5-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:13:54.926043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.926000 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:13:54.950422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.950358 4920 generic.go:356] "Generic (PLEG): container finished" podID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" containerID="4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3" exitCode=0 Oct 30 09:13:54.950910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.950484 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" Oct 30 09:13:54.950910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.950487 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" event={"ID":"d63a8edd-828b-4696-b88e-f0a384ec1fb5","Type":"ContainerDied","Data":"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3"} Oct 30 09:13:54.950910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.950550 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4" event={"ID":"d63a8edd-828b-4696-b88e-f0a384ec1fb5","Type":"ContainerDied","Data":"a337dfd9bea2116eed864a4d9fcdfc009e0e6f1b16f25b48f2d46eef6a4881d6"} Oct 30 09:13:54.950910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.950581 4920 scope.go:122] "RemoveContainer" containerID="4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3" Oct 30 09:13:54.953745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.953721 4920 generic.go:356] "Generic (PLEG): container finished" podID="15fd7d94-a7c2-4855-8256-365af8b51408" containerID="1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f" exitCode=0 Oct 30 09:13:54.953958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.953784 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" Oct 30 09:13:54.953958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.953827 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" event={"ID":"15fd7d94-a7c2-4855-8256-365af8b51408","Type":"ContainerDied","Data":"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f"} Oct 30 09:13:54.953958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.953881 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz" event={"ID":"15fd7d94-a7c2-4855-8256-365af8b51408","Type":"ContainerDied","Data":"0a84aba5bd1231b0d14ece7664ad13fb903467d49cc490b2b37e1e3e4689b596"} Oct 30 09:13:54.956258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.955997 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerStarted","Data":"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b"} Oct 30 09:13:54.956258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.956047 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerStarted","Data":"defbdeecab7e3c493067e998f2393ed13a6e6c5b791e936a8758423510404e6b"} Oct 30 09:13:54.963168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.963147 4920 scope.go:122] "RemoveContainer" containerID="4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3" Oct 30 09:13:54.963551 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:54.963522 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3\": container with ID starting with 4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3 not found: ID does not exist" containerID="4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3" Oct 30 09:13:54.963640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.963570 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3"} err="failed to get container status \"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3\": rpc error: code = NotFound desc = could not find container \"4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3\": container with ID starting with 4e17db7006cad6312b52c2bc030cd0e6b325d93e8baf0fb81bd47a49cc0c0ef3 not found: ID does not exist" Oct 30 09:13:54.963640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.963593 4920 scope.go:122] "RemoveContainer" containerID="1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f" Oct 30 09:13:54.976151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.975412 4920 scope.go:122] "RemoveContainer" containerID="1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f" Oct 30 09:13:54.976620 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:13:54.976597 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f\": container with ID starting with 1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f not found: ID does not exist" containerID="1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f" Oct 30 09:13:54.976724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:54.976626 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f"} err="failed to get container status \"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f\": rpc error: code = NotFound desc = could not find container \"1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f\": container with ID starting with 1752160d2ba7dc087d08eb984d99121bcb5561bcf84ee154c6b7573a69e9302f not found: ID does not exist" Oct 30 09:13:55.006554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.006495 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podStartSLOduration=2.006476184 podStartE2EDuration="2.006476184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:55.002661639 +0000 UTC m=+1198.726088783" watchObservedRunningTime="2026-10-30 09:13:55.006476184 +0000 UTC m=+1198.729903327" Oct 30 09:13:55.061061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.061023 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:55.062089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.061826 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:55.066397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.066372 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70374d4e-5558c99454-p6dl4"] Oct 30 09:13:55.093202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.093169 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:55.095931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.095906 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70374d4e-6f579c8559-xxthz"] Oct 30 09:13:55.150847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.150449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:55.222240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.222188 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:55.227645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.227582 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:55.966777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.966747 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:55.967806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:55.967783 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:13:56.030961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.030916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:56.125489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.125454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:56.400101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.400068 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fd7d94-a7c2-4855-8256-365af8b51408" path="/var/lib/kubelet/pods/15fd7d94-a7c2-4855-8256-365af8b51408/volumes" Oct 30 09:13:56.400677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.400653 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" path="/var/lib/kubelet/pods/d63a8edd-828b-4696-b88e-f0a384ec1fb5/volumes" Oct 30 09:13:56.557279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.557255 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:13:56.557361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.557279 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:13:56.894917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.894879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:56.994603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:56.994568 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:57.011091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011059 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 09:13:57.011615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011596 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:57.011715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011617 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8596595e-3cb5-4dce-aa66-0a507f1e6c23" containerName="istio-proxy" Oct 30 09:13:57.011715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011666 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" containerName="payload-processing" Oct 30 09:13:57.011715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011674 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" containerName="payload-processing" Oct 30 09:13:57.011715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011699 4920 cpu_manager.go:414] "containerMap: removing container" podUID="15fd7d94-a7c2-4855-8256-365af8b51408" containerName="payload-pre-processing" Oct 30 09:13:57.011715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011707 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fd7d94-a7c2-4855-8256-365af8b51408" containerName="payload-pre-processing" Oct 30 09:13:57.011948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011827 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63a8edd-828b-4696-b88e-f0a384ec1fb5" containerName="payload-processing" Oct 30 09:13:57.011948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.011842 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fd7d94-a7c2-4855-8256-365af8b51408" containerName="payload-pre-processing" Oct 30 09:13:57.017031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.017011 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.032896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.032862 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 09:13:57.039625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.039594 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2f69c8\"" Oct 30 09:13:57.039769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.039631 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2f69c8-dockercfg-mld8m\"" Oct 30 09:13:57.069114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.069085 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 09:13:57.072839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.072824 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.088397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.088368 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 09:13:57.164075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.163936 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkjn\" (UniqueName: \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.164075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.163999 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.164297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.164105 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjx2w\" (UniqueName: \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.164297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.164178 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.266042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266005 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkjn\" (UniqueName: \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.266261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266056 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.266261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266087 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjx2w\" (UniqueName: \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.266261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266138 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.266921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.267024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.266938 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.283034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.282999 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjx2w\" (UniqueName: \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\") pod \"payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.283159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.283058 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkjn\" (UniqueName: \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\") pod \"payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.329851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.329822 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.389121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.387521 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.407865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.407765 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:57.498727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.498647 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:57.639733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.639698 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 09:13:57.640334 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:57.640309 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328a0d1b_a315_43f9_9bde_c0bdc2503b27.slice/crio-ed070c644bf5c15f60308083184099445db244c7ff2624a06f8f218ff0789094 WatchSource:0}: Error finding container ed070c644bf5c15f60308083184099445db244c7ff2624a06f8f218ff0789094: Status 404 returned error can't find the container with id ed070c644bf5c15f60308083184099445db244c7ff2624a06f8f218ff0789094 Oct 30 09:13:57.709906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.709878 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 09:13:57.710596 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:13:57.710573 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff9350e_f9cd_4ed6_9b44_6305633eb32e.slice/crio-6baae6623be1814042aa07b9728491fa66ed800a2f4147722dd73eb25f4b902f WatchSource:0}: Error finding container 6baae6623be1814042aa07b9728491fa66ed800a2f4147722dd73eb25f4b902f: Status 404 returned error can't find the container with id 6baae6623be1814042aa07b9728491fa66ed800a2f4147722dd73eb25f4b902f Oct 30 09:13:57.969965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.969880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:57.985468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.985427 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" event={"ID":"5ff9350e-f9cd-4ed6-9b44-6305633eb32e","Type":"ContainerStarted","Data":"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da"} Oct 30 09:13:57.985659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.985499 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:13:57.985659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.985522 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" event={"ID":"5ff9350e-f9cd-4ed6-9b44-6305633eb32e","Type":"ContainerStarted","Data":"6baae6623be1814042aa07b9728491fa66ed800a2f4147722dd73eb25f4b902f"} Oct 30 09:13:57.987140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.987112 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" event={"ID":"328a0d1b-a315-43f9-9bde-c0bdc2503b27","Type":"ContainerStarted","Data":"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2"} Oct 30 09:13:57.987338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.987320 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:13:57.987380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:57.987365 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" event={"ID":"328a0d1b-a315-43f9-9bde-c0bdc2503b27","Type":"ContainerStarted","Data":"ed070c644bf5c15f60308083184099445db244c7ff2624a06f8f218ff0789094"} Oct 30 09:13:58.021061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.021008 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" podStartSLOduration=1.020995467 podStartE2EDuration="1.020995467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:58.019764408 +0000 UTC m=+1201.743191552" watchObservedRunningTime="2026-10-30 09:13:58.020995467 +0000 UTC m=+1201.744422610" Oct 30 09:13:58.054558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.054507 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" podStartSLOduration=2.054492224 podStartE2EDuration="2.054492224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:58.052196785 +0000 UTC m=+1201.775623929" watchObservedRunningTime="2026-10-30 09:13:58.054492224 +0000 UTC m=+1201.777919363" Oct 30 09:13:58.069997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.069965 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:58.169825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.169795 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:58.271339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.271255 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:58.378107 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.378070 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:58.655244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:58.654509 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:13:59.409519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:59.409480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:59.510382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:59.510352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:59.611534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:59.611498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:13:59.929843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:13:59.929790 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" podUID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" containerName="payload-pre-processing" containerID="cri-o://b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b" gracePeriod=25 Oct 30 09:14:00.225268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.225061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:00.251471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.251446 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:14:00.329446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.329411 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\" (UniqueName: \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\") pod \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " Oct 30 09:14:00.330484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.330456 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\") pod \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\" (UID: \"e0c97384-406a-4027-a3fd-6db6d1c1fc64\") " Oct 30 09:14:00.331356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.331332 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume" pod "e0c97384-406a-4027-a3fd-6db6d1c1fc64" (UID: "e0c97384-406a-4027-a3fd-6db6d1c1fc64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:14:00.331491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.331375 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:00.336198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.336165 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl" pod "e0c97384-406a-4027-a3fd-6db6d1c1fc64" (UID: "e0c97384-406a-4027-a3fd-6db6d1c1fc64"). InnerVolumeSpecName "kube-api-access-f46hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:14:00.350973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.350946 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp"] Oct 30 09:14:00.353495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.353468 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6cf9ebb1-zk7cp"] Oct 30 09:14:00.435267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.421671 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b907a46e-4d03-419c-a734-bb215d3504be" path="/var/lib/kubelet/pods/b907a46e-4d03-419c-a734-bb215d3504be/volumes" Oct 30 09:14:00.435802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.432656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:00.435965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.433321 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c97384-406a-4027-a3fd-6db6d1c1fc64-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:14:00.435965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.435912 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46hl\" (UniqueName: \"kubernetes.io/projected/e0c97384-406a-4027-a3fd-6db6d1c1fc64-kube-api-access-f46hl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:14:00.470077 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:14:00.469990 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c97384_406a_4027_a3fd_6db6d1c1fc64.slice\": RecentStats: unable to find data in memory cache]" Oct 30 09:14:00.543872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:00.543831 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.029015 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.028981 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" containerID="b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b" exitCode=0 Oct 30 09:14:01.029195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.029040 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" Oct 30 09:14:01.029195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.029056 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" event={"ID":"e0c97384-406a-4027-a3fd-6db6d1c1fc64","Type":"ContainerDied","Data":"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b"} Oct 30 09:14:01.029195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.029101 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g" event={"ID":"e0c97384-406a-4027-a3fd-6db6d1c1fc64","Type":"ContainerDied","Data":"39a420644a6b391605af392236cf50fb6489796214543c296af7bff3aa039544"} Oct 30 09:14:01.029195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.029122 4920 scope.go:122] "RemoveContainer" containerID="b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b" Oct 30 09:14:01.038307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.038284 4920 scope.go:122] "RemoveContainer" containerID="b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b" Oct 30 09:14:01.038618 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:14:01.038591 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b\": container with ID starting with b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b not found: ID does not exist" containerID="b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b" Oct 30 09:14:01.038693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.038626 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b"} err="failed to get container status \"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b\": rpc error: code = NotFound desc = could not find container \"b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b\": container with ID starting with b82cc9bf3c608a8c54ac8ce2c7ee3b400b1f0b0e28f309fe15a4c96be211f29b not found: ID does not exist" Oct 30 09:14:01.072641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.072613 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:14:01.076434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.076411 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6cf9ebb1-f484d5c67-lwl8g"] Oct 30 09:14:01.162955 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:14:01.162913 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:14:01.192641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.192605 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.344415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.344378 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.416237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.415684 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:14:01.416413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.416376 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" containerName="payload-pre-processing" Oct 30 09:14:01.416413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.416396 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" containerName="payload-pre-processing" Oct 30 09:14:01.416557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.416547 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" containerName="payload-pre-processing" Oct 30 09:14:01.435580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.435550 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:14:01.435972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.435687 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.448366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.448341 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-176b6c\"" Oct 30 09:14:01.448544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.448524 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-176b6c-dockercfg-g6z2j\"" Oct 30 09:14:01.456925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.456891 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69rw\" (UniqueName: \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.457029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.456970 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.476744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.476717 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.565239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.564853 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69rw\" (UniqueName: \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.565239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.564959 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.574415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.574282 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.583253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.583204 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69rw\" (UniqueName: \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\") pod \"payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.751854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.751778 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:01.832187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.832139 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:14:01.840145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.840126 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.862141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.859027 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:14:01.865905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.865859 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c5447d-openshift-default-dockercfg-nx55s\"" Oct 30 09:14:01.867515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867477 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867524 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867551 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867647 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867735 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867776 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867846 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867885 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rbb\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.867974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.867915 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.876989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.876963 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.959725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.959694 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:01.968933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.968907 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.968948 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.968978 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969011 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969039 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969107 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969182 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969387 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969427 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969443 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969471 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969507 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rbb\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969732 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.969938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.969919 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.971517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.971491 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.971660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.971642 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.985890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.985857 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:01.985997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:01.985933 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rbb\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:02.067987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.067894 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:02.095308 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:14:02.095275 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5212c0_889d_445c_ad41_7abf90a81555.slice/crio-85b9dd055f517e38ef746e0886c9fe2e45b1958c9457c99921ce18bd4a9a86b2 WatchSource:0}: Error finding container 85b9dd055f517e38ef746e0886c9fe2e45b1958c9457c99921ce18bd4a9a86b2: Status 404 returned error can't find the container with id 85b9dd055f517e38ef746e0886c9fe2e45b1958c9457c99921ce18bd4a9a86b2 Oct 30 09:14:02.095762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.095738 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:14:02.169859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.169832 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:02.403298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.397778 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c97384-406a-4027-a3fd-6db6d1c1fc64" path="/var/lib/kubelet/pods/e0c97384-406a-4027-a3fd-6db6d1c1fc64/volumes" Oct 30 09:14:02.526457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.525123 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:14:02.532337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.532048 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:02.532337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.532138 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:02.532337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.532208 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:02.989496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:02.989462 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:03.080239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.076750 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" event={"ID":"9b5212c0-889d-445c-ad41-7abf90a81555","Type":"ContainerStarted","Data":"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827"} Oct 30 09:14:03.080239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.076830 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" event={"ID":"9b5212c0-889d-445c-ad41-7abf90a81555","Type":"ContainerStarted","Data":"85b9dd055f517e38ef746e0886c9fe2e45b1958c9457c99921ce18bd4a9a86b2"} Oct 30 09:14:03.080239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.077394 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:03.080239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.079027 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerStarted","Data":"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab"} Oct 30 09:14:03.080239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.079076 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerStarted","Data":"d2ba17acb92ac622f5ca8a2329ffd0d20f0fe86b3a9d9e84b8ef01a6ef6e02bd"} Oct 30 09:14:03.130243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.128749 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" podStartSLOduration=2.128730119 podStartE2EDuration="2.128730119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:03.114720578 +0000 UTC m=+1206.838147750" watchObservedRunningTime="2026-10-30 09:14:03.128730119 +0000 UTC m=+1206.852157295" Oct 30 09:14:03.157081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.157031 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podStartSLOduration=2.157017096 podStartE2EDuration="2.157017096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:03.154948333 +0000 UTC m=+1206.878375486" watchObservedRunningTime="2026-10-30 09:14:03.157017096 +0000 UTC m=+1206.880444233" Oct 30 09:14:03.171076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.171038 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:03.176722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.176693 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:03.550380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.543149 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:03.639114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.639074 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:03.740148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:03.740056 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:04.086059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:04.086028 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:04.087009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:04.086990 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:05.125818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:05.125776 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:05.213812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:05.213746 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:05.339407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:05.339359 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:05.660792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:05.660760 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:05.760876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:05.760782 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:06.955394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:06.955351 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:14:06.955874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:06.955429 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:14:06.959565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:06.959541 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:07.864668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:07.864634 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:07.961639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:07.961610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:08.047891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.047853 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:14:08.052897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.052870 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.069683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.069654 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:14:08.105516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.105487 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c5447d-dockercfg-7nqqq\"" Oct 30 09:14:08.105802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.105752 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c5447d\"" Oct 30 09:14:08.174510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.174017 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjm8\" (UniqueName: \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.174510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.174356 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.275007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.274969 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.275187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.275026 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjm8\" (UniqueName: \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.275770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.275743 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.294396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.294359 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjm8\" (UniqueName: \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\") pod \"payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.368281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.368247 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:08.686885 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:14:08.686856 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b583ec7_6a62_4e11_9dfd_a6f4c2c06e37.slice/crio-88f9c87c4ea1faf66264f184bf10ef2921a9947d996d79ecd8e7272430a4fce2 WatchSource:0}: Error finding container 88f9c87c4ea1faf66264f184bf10ef2921a9947d996d79ecd8e7272430a4fce2: Status 404 returned error can't find the container with id 88f9c87c4ea1faf66264f184bf10ef2921a9947d996d79ecd8e7272430a4fce2 Oct 30 09:14:08.687034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.686895 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:14:08.752347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.752319 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:08.995934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.995848 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 09:14:08.995934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:08.995931 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 09:14:09.184543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.184500 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" event={"ID":"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37","Type":"ContainerStarted","Data":"3c676eb1e286c4c9005cfa316c1af72e6d80091fadf0af06f6cbabfd05c0ff49"} Oct 30 09:14:09.184706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.184576 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" event={"ID":"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37","Type":"ContainerStarted","Data":"88f9c87c4ea1faf66264f184bf10ef2921a9947d996d79ecd8e7272430a4fce2"} Oct 30 09:14:09.185517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.185497 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:09.187458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.187434 4920 generic.go:356] "Generic (PLEG): container finished" podID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerID="6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8" exitCode=0 Oct 30 09:14:09.187595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.187466 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerDied","Data":"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8"} Oct 30 09:14:09.187861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.187844 4920 scope.go:122] "RemoveContainer" containerID="6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8" Oct 30 09:14:09.189403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.189370 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:09.189519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.189450 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:09.189519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.189500 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:09.377242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:09.372320 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" podStartSLOduration=1.305209875 podStartE2EDuration="1.305209875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:09.258655433 +0000 UTC m=+1212.982082577" watchObservedRunningTime="2026-10-30 09:14:09.305209875 +0000 UTC m=+1213.028637019" Oct 30 09:14:10.211240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:10.209412 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerStarted","Data":"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743"} Oct 30 09:14:10.487740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:10.487641 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:10.721242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:10.710668 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:14:10.721242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:10.710795 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:10.721242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:10.710837 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:11.709240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:11.708809 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:11.709240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:11.708892 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:12.085305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:12.085265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:12.715709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:12.715669 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:12.716390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:12.716348 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:13.449777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:13.449741 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:13.708295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:13.707673 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:13.708295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:13.707747 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:13.814264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:13.810094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:13.910299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:13.897157 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:14.095003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.094955 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:14:14.142133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.142094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:14.279260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.273135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:14.376944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.376823 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:14.710292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.708673 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:14.710292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.708733 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:14.853326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:14.853291 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:15.690342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:15.690308 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:15.712956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:15.712510 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:14:15.712956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:15.712568 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 09:14:15.846068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:15.846031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:16.711111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:16.711076 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:14:16.711670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:16.711498 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:14:16.712320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:16.712299 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:14:17.289143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:17.289109 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:17.451987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:17.451953 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:17.555084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:17.555005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:17.849179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:17.849095 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:18.316145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.316110 4920 generic.go:356] "Generic (PLEG): container finished" podID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerID="1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3" exitCode=0 Oct 30 09:14:18.316366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.316148 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerDied","Data":"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3"} Oct 30 09:14:18.316671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.316649 4920 scope.go:122] "RemoveContainer" containerID="1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3" Oct 30 09:14:18.318286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.318254 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:18.318377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.318338 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:18.318421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.318381 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:18.431680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.431134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:18.542067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.541858 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:18.682586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:18.682527 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:19.174161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.174124 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:19.359433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.359395 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerStarted","Data":"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05"} Oct 30 09:14:19.362522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.362496 4920 generic.go:356] "Generic (PLEG): container finished" podID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerID="bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b" exitCode=0 Oct 30 09:14:19.362669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.362568 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerDied","Data":"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b"} Oct 30 09:14:19.362961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.362943 4920 scope.go:122] "RemoveContainer" containerID="bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b" Oct 30 09:14:19.364097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.364068 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:19.364185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.364148 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:19.364276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:19.364194 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:20.061802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:20.061766 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:14:20.062700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:20.062672 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:14:20.062899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:20.062876 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 09:14:20.382941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:20.382860 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerStarted","Data":"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0"} Oct 30 09:14:20.894447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:20.894413 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:21.068240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.061712 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:14:21.068240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.061784 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 09:14:21.225298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.223547 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:14:21.225298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.223686 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 09:14:21.225298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.223729 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 09:14:21.250413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.250382 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:14:21.932275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:21.929376 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:14:22.068242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.067732 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:14:22.068242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.067807 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 09:14:22.161204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.161162 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:22.224757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.224662 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 09:14:22.225051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.225020 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 09:14:22.574933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:22.574836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:23.065660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:23.065623 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:14:23.066400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:23.066368 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 09:14:23.225455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:23.225419 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 09:14:23.225737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:23.225710 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 09:14:24.065711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.065675 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 09:14:24.066305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.066273 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 09:14:24.183579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.183426 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:24.223070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.222618 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 09:14:24.223070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.222690 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 09:14:24.842722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.842684 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:24.940937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:24.940882 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:25.227740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.227633 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 09:14:25.227740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.227705 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 09:14:25.246853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.246825 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:14:25.248500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.248475 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:14:25.809293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.809240 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:25.967668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:25.966657 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:14:26.007243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:26.007141 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:26.228089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:26.228010 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:14:26.798927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:26.798896 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:14:27.133679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.133587 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:14:27.139537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.139512 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.154247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.154208 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"simulator-w2-e3d2ff-kserve-self-signed-certs\"" Oct 30 09:14:27.155143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.155113 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"default-dockercfg-jsvq8\"" Oct 30 09:14:27.155267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.155254 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"openshift-service-ca.crt\"" Oct 30 09:14:27.155350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.155333 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"kube-root-ca.crt\"" Oct 30 09:14:27.155408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.155364 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:14:27.172171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.172099 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:27.196883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.196851 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.197043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.196897 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8jf\" (UniqueName: \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.197043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.196961 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.197043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.197025 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.197146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.197070 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.197184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.197140 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.298607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298568 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.298607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298620 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298678 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298740 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298782 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8jf\" (UniqueName: \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.298816 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.299075 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.299344 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.299589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.299568 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.301424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.301398 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.301925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.301902 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.317523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.317495 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8jf\" (UniqueName: \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\") pod \"simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.459516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.459416 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:27.579443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.579379 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:27.767296 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:14:27.767191 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6937d716_3d86_492b_b48a_42e96fbe40d4.slice/crio-e168d6e667d990a91dbfb05661563ba1c1e475fa849ead9771bcc2dfb055e3f4 WatchSource:0}: Error finding container e168d6e667d990a91dbfb05661563ba1c1e475fa849ead9771bcc2dfb055e3f4: Status 404 returned error can't find the container with id e168d6e667d990a91dbfb05661563ba1c1e475fa849ead9771bcc2dfb055e3f4 Oct 30 09:14:27.768259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.768234 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:14:27.807547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:27.807508 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" event={"ID":"6937d716-3d86-492b-b48a-42e96fbe40d4","Type":"ContainerStarted","Data":"e168d6e667d990a91dbfb05661563ba1c1e475fa849ead9771bcc2dfb055e3f4"} Oct 30 09:14:28.815205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.815120 4920 generic.go:356] "Generic (PLEG): container finished" podID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerID="3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab" exitCode=0 Oct 30 09:14:28.815741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.815198 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerDied","Data":"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab"} Oct 30 09:14:28.815741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.815680 4920 scope.go:122] "RemoveContainer" containerID="3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab" Oct 30 09:14:28.816803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.816782 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" event={"ID":"6937d716-3d86-492b-b48a-42e96fbe40d4","Type":"ContainerStarted","Data":"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07"} Oct 30 09:14:28.816983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.816948 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:28.817098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.817020 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:28.817098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.817056 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:28.906597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:28.906551 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podStartSLOduration=1.9065353329999999 podStartE2EDuration="1.906535333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:28.904868137 +0000 UTC m=+1232.628295297" watchObservedRunningTime="2026-10-30 09:14:28.906535333 +0000 UTC m=+1232.629962474" Oct 30 09:14:29.153926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:29.153847 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:29.460119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:29.460039 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:29.836620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:29.836583 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerStarted","Data":"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd"} Oct 30 09:14:30.170717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.170638 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:30.172050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.172022 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:14:30.172153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.172074 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 09:14:30.467102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.467020 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:30.726294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.726049 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:30.933054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:30.933011 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:31.175292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:31.175244 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:14:31.175579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:31.175538 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 09:14:31.176411 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:14:31.176244 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:14:31.577899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:31.577863 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:31.672710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:31.672679 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:31.773735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:31.773696 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:32.170985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:32.170953 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:14:32.171423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:32.171007 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 09:14:32.536884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:32.536845 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:33.170472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:33.170429 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:14:33.170689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:33.170494 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 09:14:33.963813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:33.963779 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:34.086818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:34.086783 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:34.170447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:34.170406 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:14:34.170660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:34.170477 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 09:14:35.174842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:35.174811 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:35.176011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:35.175988 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:14:35.451724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:35.451640 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:37.044802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:37.044767 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:37.460413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:37.460324 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:37.473526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:37.473500 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:37.929751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:37.929718 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:14:38.643963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:38.643934 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:40.165860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:40.165827 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:40.685585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:40.685547 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:41.641950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:41.641917 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:43.122985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.122949 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:43.383042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.382947 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:14:43.388695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.388666 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.404553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.404524 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:14:43.413553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.413511 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"openshift-service-ca.crt\"" Oct 30 09:14:43.413739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.413595 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"kube-root-ca.crt\"" Oct 30 09:14:43.413820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.413728 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"simulator-w1-c5447d-kserve-self-signed-certs\"" Oct 30 09:14:43.415893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.415859 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"default-dockercfg-9qf68\"" Oct 30 09:14:43.431971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.431940 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:43.536629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536597 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.536800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536740 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.536800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536768 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.536875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536801 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kxn\" (UniqueName: \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.536875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536850 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.536945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.536885 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638031 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638098 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638130 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kxn\" (UniqueName: \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638175 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638247 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638337 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.638809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.638785 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.639456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.639432 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.639697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.639679 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.641764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.641746 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.642350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.642333 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.658106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.658072 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kxn\" (UniqueName: \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\") pod \"simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:43.703302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:43.703271 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:44.002474 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:14:44.002447 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod297160b5_6012_4f00_81c8_c01911769694.slice/crio-164cad662e9e33702079120a5140bdbf71bccccfa9bc26c4591b824df056b312 WatchSource:0}: Error finding container 164cad662e9e33702079120a5140bdbf71bccccfa9bc26c4591b824df056b312: Status 404 returned error can't find the container with id 164cad662e9e33702079120a5140bdbf71bccccfa9bc26c4591b824df056b312 Oct 30 09:14:44.005168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:44.005141 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:14:44.639940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:44.639901 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:44.968370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:44.968280 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" event={"ID":"297160b5-6012-4f00-81c8-c01911769694","Type":"ContainerStarted","Data":"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de"} Oct 30 09:14:44.968370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:44.968339 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" event={"ID":"297160b5-6012-4f00-81c8-c01911769694","Type":"ContainerStarted","Data":"164cad662e9e33702079120a5140bdbf71bccccfa9bc26c4591b824df056b312"} Oct 30 09:14:45.006336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:45.006280 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podStartSLOduration=2.006262804 podStartE2EDuration="2.006262804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:45.004659912 +0000 UTC m=+1248.728087056" watchObservedRunningTime="2026-10-30 09:14:45.006262804 +0000 UTC m=+1248.729689947" Oct 30 09:14:45.704260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:45.704205 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:46.286883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:46.286829 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:47.838970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:47.838899 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:48.005443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:48.005409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:49.421082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:49.421036 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:50.594820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:50.594783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:50.870941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:50.870849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:52.315673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:52.315636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:53.335756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:53.335718 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:14:53.703690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:53.703603 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:53.717212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:53.717181 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:53.874949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:53.874909 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:54.043000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:54.042974 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:14:55.449793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:55.449750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:57.039639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:57.039612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:58.636941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:58.636908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:14:59.651596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:14:59.651564 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:00.127743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.127708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:00.150983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.150955 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:00.156249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.156227 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:00.156405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.156338 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.160163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.160140 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:00.160307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.160287 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.164608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.164585 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:00.164724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.164656 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.168176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.168156 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.172573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.172550 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:15:00.176139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.176119 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:00.182721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.182697 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:00.184394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.184373 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:00.197501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.197470 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:15:00.197757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.197607 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:15:00.206253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.205767 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:00.228733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.228702 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxr2d\" (UniqueName: \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.228881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.228756 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wtjp\" (UniqueName: \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.228946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.228914 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.229005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.228974 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zt7\" (UniqueName: \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.229066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.229026 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.229142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.229129 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.229199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.229169 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.229279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.229199 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnf79\" (UniqueName: \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.330173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330139 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.330370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330186 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zt7\" (UniqueName: \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.330370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330241 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.330370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330276 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.330370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330298 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.330617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330406 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnf79\" (UniqueName: \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.330617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330467 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxr2d\" (UniqueName: \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.330617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330512 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtjp\" (UniqueName: \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.330936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.330913 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.331090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.331070 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.331154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.331119 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.331289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.331271 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.348415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.348386 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zt7\" (UniqueName: \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\") pod \"maas-api-key-cleanup-29889195-cg8d2\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.348655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.348630 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxr2d\" (UniqueName: \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.348943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.348923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtjp\" (UniqueName: \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.349192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.349172 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnf79\" (UniqueName: \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.473795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.473699 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:00.502928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.502896 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:00.516162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.516130 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:00.522368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.522347 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:00.778975 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:00.778949 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1370b934_2bba_435c_9db6_ed4655c88448.slice/crio-e25ed19ad38d6ecf339813ecf2f08fc5f077af51c64fd0cf60b14fa1eae6aeb9 WatchSource:0}: Error finding container e25ed19ad38d6ecf339813ecf2f08fc5f077af51c64fd0cf60b14fa1eae6aeb9: Status 404 returned error can't find the container with id e25ed19ad38d6ecf339813ecf2f08fc5f077af51c64fd0cf60b14fa1eae6aeb9 Oct 30 09:15:00.780265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.780237 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:00.832917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.832890 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:00.835109 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:00.835085 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b220a92_0d46_4e9d_8341_33d67bb05173.slice/crio-90ba06b8db178e1287e600831b2998f5e9796b7eba6a9b7c7cad09bd43bfd665 WatchSource:0}: Error finding container 90ba06b8db178e1287e600831b2998f5e9796b7eba6a9b7c7cad09bd43bfd665: Status 404 returned error can't find the container with id 90ba06b8db178e1287e600831b2998f5e9796b7eba6a9b7c7cad09bd43bfd665 Oct 30 09:15:00.844155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.844122 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:00.845554 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:00.845528 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef976272_a7b6_4c88_bf10_6d69cfd8afd7.slice/crio-50b3560922a320df6cdbd6895790681e1c97d7150cf0857e826c9ddca5566cfa WatchSource:0}: Error finding container 50b3560922a320df6cdbd6895790681e1c97d7150cf0857e826c9ddca5566cfa: Status 404 returned error can't find the container with id 50b3560922a320df6cdbd6895790681e1c97d7150cf0857e826c9ddca5566cfa Oct 30 09:15:00.874191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:00.872735 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:00.881728 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:00.881700 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e7ac88_b8b7_439d_86ee_d597f83dc08b.slice/crio-881a31aa7cd022141bd61cf66f48649b97fda0a70d3cf4ab1ac9565af3bb148f WatchSource:0}: Error finding container 881a31aa7cd022141bd61cf66f48649b97fda0a70d3cf4ab1ac9565af3bb148f: Status 404 returned error can't find the container with id 881a31aa7cd022141bd61cf66f48649b97fda0a70d3cf4ab1ac9565af3bb148f Oct 30 09:15:01.097275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:01.097209 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerStarted","Data":"90ba06b8db178e1287e600831b2998f5e9796b7eba6a9b7c7cad09bd43bfd665"} Oct 30 09:15:01.099059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:01.099028 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" event={"ID":"e9e7ac88-b8b7-439d-86ee-d597f83dc08b","Type":"ContainerStarted","Data":"881a31aa7cd022141bd61cf66f48649b97fda0a70d3cf4ab1ac9565af3bb148f"} Oct 30 09:15:01.101423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:01.101397 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerStarted","Data":"e25ed19ad38d6ecf339813ecf2f08fc5f077af51c64fd0cf60b14fa1eae6aeb9"} Oct 30 09:15:01.102757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:01.102732 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerStarted","Data":"50b3560922a320df6cdbd6895790681e1c97d7150cf0857e826c9ddca5566cfa"} Oct 30 09:15:01.199410 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:01.199374 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:15:02.040590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.040559 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:02.120814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.120776 4920 generic.go:356] "Generic (PLEG): container finished" podID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerID="a01b87c2b448e1060679be9b75b32f8e2d2502e63ae96ca9471d9c1696cfd4d6" exitCode=60 Oct 30 09:15:02.121208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.121181 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerDied","Data":"a01b87c2b448e1060679be9b75b32f8e2d2502e63ae96ca9471d9c1696cfd4d6"} Oct 30 09:15:02.121405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.121387 4920 scope.go:122] "RemoveContainer" containerID="a01b87c2b448e1060679be9b75b32f8e2d2502e63ae96ca9471d9c1696cfd4d6" Oct 30 09:15:02.123250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.123202 4920 generic.go:356] "Generic (PLEG): container finished" podID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerID="4c9b285bb4114201cb95301ddb66b1216dbeeba6a6281baddd9457a2012baae9" exitCode=60 Oct 30 09:15:02.123342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.123251 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" event={"ID":"e9e7ac88-b8b7-439d-86ee-d597f83dc08b","Type":"ContainerDied","Data":"4c9b285bb4114201cb95301ddb66b1216dbeeba6a6281baddd9457a2012baae9"} Oct 30 09:15:02.123540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.123514 4920 scope.go:122] "RemoveContainer" containerID="4c9b285bb4114201cb95301ddb66b1216dbeeba6a6281baddd9457a2012baae9" Oct 30 09:15:02.126175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.126154 4920 generic.go:356] "Generic (PLEG): container finished" podID="1370b934-2bba-435c-9db6-ed4655c88448" containerID="479e92be3e00883592bfe150ead1308a23312eb2c03ab4e14b80a4c641c990c0" exitCode=60 Oct 30 09:15:02.126319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.126264 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerDied","Data":"479e92be3e00883592bfe150ead1308a23312eb2c03ab4e14b80a4c641c990c0"} Oct 30 09:15:02.126520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.126473 4920 scope.go:122] "RemoveContainer" containerID="479e92be3e00883592bfe150ead1308a23312eb2c03ab4e14b80a4c641c990c0" Oct 30 09:15:02.128386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.128364 4920 generic.go:356] "Generic (PLEG): container finished" podID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerID="debaff1c207099b433a0cca19c1a2e200693c975b1ef58c4268f2afe5e4576ad" exitCode=60 Oct 30 09:15:02.128467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.128410 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerDied","Data":"debaff1c207099b433a0cca19c1a2e200693c975b1ef58c4268f2afe5e4576ad"} Oct 30 09:15:02.128642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:02.128627 4920 scope.go:122] "RemoveContainer" containerID="debaff1c207099b433a0cca19c1a2e200693c975b1ef58c4268f2afe5e4576ad" Oct 30 09:15:03.137280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.137240 4920 generic.go:356] "Generic (PLEG): container finished" podID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerID="4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e" exitCode=60 Oct 30 09:15:03.137820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.137328 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" event={"ID":"e9e7ac88-b8b7-439d-86ee-d597f83dc08b","Type":"ContainerDied","Data":"4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e"} Oct 30 09:15:03.137820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.137400 4920 scope.go:122] "RemoveContainer" containerID="4c9b285bb4114201cb95301ddb66b1216dbeeba6a6281baddd9457a2012baae9" Oct 30 09:15:03.137820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.137714 4920 scope.go:122] "RemoveContainer" containerID="4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e" Oct 30 09:15:03.138016 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:03.137997 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889195-cg8d2_odh-ai-gateway-infra(e9e7ac88-b8b7-439d-86ee-d597f83dc08b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" Oct 30 09:15:03.139666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.139640 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerStarted","Data":"39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417"} Oct 30 09:15:03.141666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.141642 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerStarted","Data":"b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb"} Oct 30 09:15:03.144366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.144343 4920 generic.go:356] "Generic (PLEG): container finished" podID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerID="43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4" exitCode=60 Oct 30 09:15:03.144479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.144445 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerDied","Data":"43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4"} Oct 30 09:15:03.144974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.144657 4920 scope.go:122] "RemoveContainer" containerID="43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4" Oct 30 09:15:03.144974 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:03.144925 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5_odh-ai-gateway-infra(6b220a92-0d46-4e9d-8341-33d67bb05173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" Oct 30 09:15:03.213998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.213948 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" podStartSLOduration=3.213932709 podStartE2EDuration="3.213932709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:03.210652993 +0000 UTC m=+1266.934080154" watchObservedRunningTime="2026-10-30 09:15:03.213932709 +0000 UTC m=+1266.937359852" Oct 30 09:15:03.282879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.282833 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" podStartSLOduration=3.2828197550000002 podStartE2EDuration="3.282819755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:03.281900208 +0000 UTC m=+1267.005327353" watchObservedRunningTime="2026-10-30 09:15:03.282819755 +0000 UTC m=+1267.006246912" Oct 30 09:15:03.335649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.335616 4920 scope.go:122] "RemoveContainer" containerID="a01b87c2b448e1060679be9b75b32f8e2d2502e63ae96ca9471d9c1696cfd4d6" Oct 30 09:15:03.534830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:03.534794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:04.153131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.153096 4920 generic.go:356] "Generic (PLEG): container finished" podID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerID="b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb" exitCode=60 Oct 30 09:15:04.153603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.153176 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerDied","Data":"b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb"} Oct 30 09:15:04.153603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.153260 4920 scope.go:122] "RemoveContainer" containerID="debaff1c207099b433a0cca19c1a2e200693c975b1ef58c4268f2afe5e4576ad" Oct 30 09:15:04.153603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.153460 4920 scope.go:122] "RemoveContainer" containerID="b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb" Oct 30 09:15:04.153743 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:04.153721 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c_odh-ai-gateway-infra(ef976272-a7b6-4c88-bf10-6d69cfd8afd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" Oct 30 09:15:04.162233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.155470 4920 scope.go:122] "RemoveContainer" containerID="43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4" Oct 30 09:15:04.162233 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:04.155936 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5_odh-ai-gateway-infra(6b220a92-0d46-4e9d-8341-33d67bb05173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" Oct 30 09:15:04.162233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.157764 4920 scope.go:122] "RemoveContainer" containerID="4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e" Oct 30 09:15:04.162233 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:04.158120 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889195-cg8d2_odh-ai-gateway-infra(e9e7ac88-b8b7-439d-86ee-d597f83dc08b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" Oct 30 09:15:04.164418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.164392 4920 generic.go:356] "Generic (PLEG): container finished" podID="1370b934-2bba-435c-9db6-ed4655c88448" containerID="39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417" exitCode=60 Oct 30 09:15:04.164533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.164457 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerDied","Data":"39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417"} Oct 30 09:15:04.164726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.164712 4920 scope.go:122] "RemoveContainer" containerID="39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417" Oct 30 09:15:04.164929 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:04.164911 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r_odh-ai-gateway-infra(1370b934-2bba-435c-9db6-ed4655c88448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" podUID="1370b934-2bba-435c-9db6-ed4655c88448" Oct 30 09:15:04.173968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.173948 4920 scope.go:122] "RemoveContainer" containerID="479e92be3e00883592bfe150ead1308a23312eb2c03ab4e14b80a4c641c990c0" Oct 30 09:15:04.672679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:04.672645 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:15:05.173026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:05.172994 4920 scope.go:122] "RemoveContainer" containerID="39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417" Oct 30 09:15:05.173536 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:05.173353 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r_odh-ai-gateway-infra(1370b934-2bba-435c-9db6-ed4655c88448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" podUID="1370b934-2bba-435c-9db6-ed4655c88448" Oct 30 09:15:05.174650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:05.174621 4920 scope.go:122] "RemoveContainer" containerID="b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb" Oct 30 09:15:05.174865 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:05.174842 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c_odh-ai-gateway-infra(ef976272-a7b6-4c88-bf10-6d69cfd8afd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" Oct 30 09:15:05.184168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:05.184144 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:06.954834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:06.954800 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:07.337949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:07.337912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:08.495187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:08.495155 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:09.955203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:09.955162 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:11.342724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:11.342688 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:11.555754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:11.555719 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:13.097511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:13.097456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:14.390640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:14.390600 4920 scope.go:122] "RemoveContainer" containerID="b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb" Oct 30 09:15:14.391120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:14.390683 4920 scope.go:122] "RemoveContainer" containerID="4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e" Oct 30 09:15:14.625977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:14.625937 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:15.253313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.253277 4920 generic.go:356] "Generic (PLEG): container finished" podID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerID="712dd4fe48a6b85b66aeaa4ffd7e56e2e96228f67dce186e0da8b0b880699512" exitCode=60 Oct 30 09:15:15.253494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.253346 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" event={"ID":"e9e7ac88-b8b7-439d-86ee-d597f83dc08b","Type":"ContainerDied","Data":"712dd4fe48a6b85b66aeaa4ffd7e56e2e96228f67dce186e0da8b0b880699512"} Oct 30 09:15:15.253494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.253401 4920 scope.go:122] "RemoveContainer" containerID="4d2bb8b2e8f0c925c0fb8ab791625f6539f75e03e1a930267c9cb464c83bf40e" Oct 30 09:15:15.253797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.253778 4920 scope.go:122] "RemoveContainer" containerID="712dd4fe48a6b85b66aeaa4ffd7e56e2e96228f67dce186e0da8b0b880699512" Oct 30 09:15:15.254071 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:15.254050 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889195-cg8d2_odh-ai-gateway-infra(e9e7ac88-b8b7-439d-86ee-d597f83dc08b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" Oct 30 09:15:15.255428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.255407 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerStarted","Data":"3987b9f710e13fbfd6f8dc75d8771c508fa9845c41da11720998d81a0003ada8"} Oct 30 09:15:15.377298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:15.377264 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:16.114873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.114837 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:16.263886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.263850 4920 generic.go:356] "Generic (PLEG): container finished" podID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerID="3987b9f710e13fbfd6f8dc75d8771c508fa9845c41da11720998d81a0003ada8" exitCode=60 Oct 30 09:15:16.264050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.263921 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerDied","Data":"3987b9f710e13fbfd6f8dc75d8771c508fa9845c41da11720998d81a0003ada8"} Oct 30 09:15:16.264050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.263976 4920 scope.go:122] "RemoveContainer" containerID="b51d32e76866f0c8cfac37b01fc6a28eba3f9fe45925d6292293f47f2e5559bb" Oct 30 09:15:16.264372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.264347 4920 scope.go:122] "RemoveContainer" containerID="3987b9f710e13fbfd6f8dc75d8771c508fa9845c41da11720998d81a0003ada8" Oct 30 09:15:16.264694 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:16.264673 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c_odh-ai-gateway-infra(ef976272-a7b6-4c88-bf10-6d69cfd8afd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" Oct 30 09:15:16.300937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.300912 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:16.340198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.340111 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:16.395938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.395869 4920 scope.go:122] "RemoveContainer" containerID="39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417" Oct 30 09:15:16.464014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.463975 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:16.615755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.615724 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\") pod \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " Oct 30 09:15:16.615923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.615905 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\" (UniqueName: \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\") pod \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\" (UID: \"e9e7ac88-b8b7-439d-86ee-d597f83dc08b\") " Oct 30 09:15:16.616063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.616041 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca" pod "e9e7ac88-b8b7-439d-86ee-d597f83dc08b" (UID: "e9e7ac88-b8b7-439d-86ee-d597f83dc08b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:16.616280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.616267 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:16.617817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.617800 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7" pod "e9e7ac88-b8b7-439d-86ee-d597f83dc08b" (UID: "e9e7ac88-b8b7-439d-86ee-d597f83dc08b"). InnerVolumeSpecName "kube-api-access-c9zt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:16.716961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:16.716934 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9zt7\" (UniqueName: \"kubernetes.io/projected/e9e7ac88-b8b7-439d-86ee-d597f83dc08b-kube-api-access-c9zt7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:17.273841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.273812 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" Oct 30 09:15:17.274327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.273807 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2" event={"ID":"e9e7ac88-b8b7-439d-86ee-d597f83dc08b","Type":"ContainerDied","Data":"881a31aa7cd022141bd61cf66f48649b97fda0a70d3cf4ab1ac9565af3bb148f"} Oct 30 09:15:17.274327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.274001 4920 scope.go:122] "RemoveContainer" containerID="712dd4fe48a6b85b66aeaa4ffd7e56e2e96228f67dce186e0da8b0b880699512" Oct 30 09:15:17.276072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.276044 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerStarted","Data":"b5c30c0b831b80b0d8952b50a4f52bfb0903536d38295a0ec9539d4a720b2f2e"} Oct 30 09:15:17.395231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.395186 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:17.398462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.398425 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889195-cg8d2"] Oct 30 09:15:17.438682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.438651 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:17.526719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.526619 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\" (UniqueName: \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\") pod \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " Oct 30 09:15:17.526882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.526771 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\") pod \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\" (UID: \"ef976272-a7b6-4c88-bf10-6d69cfd8afd7\") " Oct 30 09:15:17.527184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.527152 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca" pod "ef976272-a7b6-4c88-bf10-6d69cfd8afd7" (UID: "ef976272-a7b6-4c88-bf10-6d69cfd8afd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:17.528703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.528677 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp" pod "ef976272-a7b6-4c88-bf10-6d69cfd8afd7" (UID: "ef976272-a7b6-4c88-bf10-6d69cfd8afd7"). InnerVolumeSpecName "kube-api-access-2wtjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:17.627928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.627888 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:17.627928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.627919 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wtjp\" (UniqueName: \"kubernetes.io/projected/ef976272-a7b6-4c88-bf10-6d69cfd8afd7-kube-api-access-2wtjp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:17.685247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:17.685186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:18.288089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.288056 4920 generic.go:356] "Generic (PLEG): container finished" podID="1370b934-2bba-435c-9db6-ed4655c88448" containerID="b5c30c0b831b80b0d8952b50a4f52bfb0903536d38295a0ec9539d4a720b2f2e" exitCode=60 Oct 30 09:15:18.288600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.288137 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerDied","Data":"b5c30c0b831b80b0d8952b50a4f52bfb0903536d38295a0ec9539d4a720b2f2e"} Oct 30 09:15:18.288600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.288199 4920 scope.go:122] "RemoveContainer" containerID="39267a37494688eb0089b1dcd92e4f2ad54cd0868023c18465515ced192d0417" Oct 30 09:15:18.288600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.288570 4920 scope.go:122] "RemoveContainer" containerID="b5c30c0b831b80b0d8952b50a4f52bfb0903536d38295a0ec9539d4a720b2f2e" Oct 30 09:15:18.288875 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:18.288848 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r_odh-ai-gateway-infra(1370b934-2bba-435c-9db6-ed4655c88448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" podUID="1370b934-2bba-435c-9db6-ed4655c88448" Oct 30 09:15:18.289613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.289590 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" event={"ID":"ef976272-a7b6-4c88-bf10-6d69cfd8afd7","Type":"ContainerDied","Data":"50b3560922a320df6cdbd6895790681e1c97d7150cf0857e826c9ddca5566cfa"} Oct 30 09:15:18.289688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.289611 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c" Oct 30 09:15:18.304891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.304864 4920 scope.go:122] "RemoveContainer" containerID="3987b9f710e13fbfd6f8dc75d8771c508fa9845c41da11720998d81a0003ada8" Oct 30 09:15:18.361096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.361063 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:18.363992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.363969 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889195-wp48c"] Oct 30 09:15:18.370314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.370288 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:18.390405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.390378 4920 scope.go:122] "RemoveContainer" containerID="43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4" Oct 30 09:15:18.398038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.398013 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" path="/var/lib/kubelet/pods/e9e7ac88-b8b7-439d-86ee-d597f83dc08b/volumes" Oct 30 09:15:18.398511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:18.398490 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" path="/var/lib/kubelet/pods/ef976272-a7b6-4c88-bf10-6d69cfd8afd7/volumes" Oct 30 09:15:19.242625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.242579 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:19.328174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.328144 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerStarted","Data":"976c427db4dfb5a25e4c01f923b6b79bc5ed1cec83451ad3ce46fbd2a984c5c0"} Oct 30 09:15:19.400376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.399822 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" podStartSLOduration=19.399802133 podStartE2EDuration="19.399802133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:19.39676464 +0000 UTC m=+1283.120191783" watchObservedRunningTime="2026-10-30 09:15:19.399802133 +0000 UTC m=+1283.123229278" Oct 30 09:15:19.483304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.483274 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:19.650535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.650498 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\" (UniqueName: \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\") pod \"1370b934-2bba-435c-9db6-ed4655c88448\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " Oct 30 09:15:19.650724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.650590 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\") pod \"1370b934-2bba-435c-9db6-ed4655c88448\" (UID: \"1370b934-2bba-435c-9db6-ed4655c88448\") " Oct 30 09:15:19.650951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.650928 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca" pod "1370b934-2bba-435c-9db6-ed4655c88448" (UID: "1370b934-2bba-435c-9db6-ed4655c88448"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:19.651055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.651042 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1370b934-2bba-435c-9db6-ed4655c88448-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:19.652687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.652648 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79" pod "1370b934-2bba-435c-9db6-ed4655c88448" (UID: "1370b934-2bba-435c-9db6-ed4655c88448"). InnerVolumeSpecName "kube-api-access-xnf79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:19.751658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:19.751618 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnf79\" (UniqueName: \"kubernetes.io/projected/1370b934-2bba-435c-9db6-ed4655c88448-kube-api-access-xnf79\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:20.138270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.138235 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:15:20.335874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.335846 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" Oct 30 09:15:20.336378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.335879 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r" event={"ID":"1370b934-2bba-435c-9db6-ed4655c88448","Type":"ContainerDied","Data":"e25ed19ad38d6ecf339813ecf2f08fc5f077af51c64fd0cf60b14fa1eae6aeb9"} Oct 30 09:15:20.336378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.335944 4920 scope.go:122] "RemoveContainer" containerID="b5c30c0b831b80b0d8952b50a4f52bfb0903536d38295a0ec9539d4a720b2f2e" Oct 30 09:15:20.338073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.338050 4920 generic.go:356] "Generic (PLEG): container finished" podID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerID="976c427db4dfb5a25e4c01f923b6b79bc5ed1cec83451ad3ce46fbd2a984c5c0" exitCode=60 Oct 30 09:15:20.338202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.338122 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerDied","Data":"976c427db4dfb5a25e4c01f923b6b79bc5ed1cec83451ad3ce46fbd2a984c5c0"} Oct 30 09:15:20.338567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.338550 4920 scope.go:122] "RemoveContainer" containerID="976c427db4dfb5a25e4c01f923b6b79bc5ed1cec83451ad3ce46fbd2a984c5c0" Oct 30 09:15:20.338895 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:20.338867 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5_odh-ai-gateway-infra(6b220a92-0d46-4e9d-8341-33d67bb05173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" Oct 30 09:15:20.347704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.347677 4920 scope.go:122] "RemoveContainer" containerID="43049289f90203988365cb7e6fe946bfe6bd53a6de78fc7ff4fb329224270fd4" Oct 30 09:15:20.411055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.410970 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:20.424417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.424366 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:20.434739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.434706 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f69c8-29889195-6dv2r"] Oct 30 09:15:20.779059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:20.778928 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:21.531330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.531303 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:21.675345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.675265 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\") pod \"6b220a92-0d46-4e9d-8341-33d67bb05173\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " Oct 30 09:15:21.675345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.675305 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\" (UniqueName: \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\") pod \"6b220a92-0d46-4e9d-8341-33d67bb05173\" (UID: \"6b220a92-0d46-4e9d-8341-33d67bb05173\") " Oct 30 09:15:21.675694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.675669 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca" pod "6b220a92-0d46-4e9d-8341-33d67bb05173" (UID: "6b220a92-0d46-4e9d-8341-33d67bb05173"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:21.677475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.677452 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d" pod "6b220a92-0d46-4e9d-8341-33d67bb05173" (UID: "6b220a92-0d46-4e9d-8341-33d67bb05173"). InnerVolumeSpecName "kube-api-access-wxr2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:21.776884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.776849 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b220a92-0d46-4e9d-8341-33d67bb05173-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:21.776884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:21.776876 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxr2d\" (UniqueName: \"kubernetes.io/projected/6b220a92-0d46-4e9d-8341-33d67bb05173-kube-api-access-wxr2d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:22.350231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.350186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:22.362849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.362817 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" event={"ID":"6b220a92-0d46-4e9d-8341-33d67bb05173","Type":"ContainerDied","Data":"90ba06b8db178e1287e600831b2998f5e9796b7eba6a9b7c7cad09bd43bfd665"} Oct 30 09:15:22.362849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.362841 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5" Oct 30 09:15:22.363021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.362879 4920 scope.go:122] "RemoveContainer" containerID="976c427db4dfb5a25e4c01f923b6b79bc5ed1cec83451ad3ce46fbd2a984c5c0" Oct 30 09:15:22.397653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.397619 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1370b934-2bba-435c-9db6-ed4655c88448" path="/var/lib/kubelet/pods/1370b934-2bba-435c-9db6-ed4655c88448/volumes" Oct 30 09:15:22.431007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.430902 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:22.433407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:22.433379 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889195-z4md5"] Oct 30 09:15:23.952631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:23.952600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:24.398498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:24.398462 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" path="/var/lib/kubelet/pods/6b220a92-0d46-4e9d-8341-33d67bb05173/volumes" Oct 30 09:15:25.443505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:25.443467 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:26.945397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:26.945365 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:28.476233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:28.476191 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:30.129688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:30.129294 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:31.134686 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:15:31.134647 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:15:31.879145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:31.879112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:33.424125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:33.424093 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:34.952358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:34.952326 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:36.518679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:36.518617 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:37.436887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:37.436851 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:15:38.059004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:38.058965 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:39.871554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:39.871507 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:41.480561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:41.480531 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:42.958581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:42.958548 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:44.413689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:44.413646 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:46.056452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:46.056407 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:47.573168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:47.573122 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:48.121987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:48.121952 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:49.196509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.196469 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:49.796298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796261 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6"] Oct 30 09:15:49.796756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796742 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796757 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796775 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796780 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796793 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796798 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796804 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.796810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796808 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796817 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796822 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796831 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796836 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796851 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796855 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796867 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796871 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796878 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796883 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796966 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796978 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796989 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.796999 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797005 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797011 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797018 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797025 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797031 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e7ac88-b8b7-439d-86ee-d597f83dc08b" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797037 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:49.797052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.797044 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:49.800652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.800633 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:49.818471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.818442 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6"] Oct 30 09:15:49.826353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.826327 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:15:49.826499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.826364 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:15:49.826550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.826502 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:15:49.827458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.827424 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:15:49.904666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.904631 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:49.904869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.904748 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:49.904869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:49.904809 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.005760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.005708 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.005978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.005818 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.005978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.005884 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.006647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.006608 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.008413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.008377 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.024148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.024121 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\") pod \"maas-api-revoke-sub-67a1d587-4hwp6\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.113988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.113898 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:50.416603 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:50.416562 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a70b0e9_a169_4ee5_9cd0_4872a7c90fc2.slice/crio-063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445 WatchSource:0}: Error finding container 063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445: Status 404 returned error can't find the container with id 063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445 Oct 30 09:15:50.417086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.417052 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6"] Oct 30 09:15:50.573935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.573891 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" event={"ID":"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2","Type":"ContainerStarted","Data":"063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445"} Oct 30 09:15:50.851896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:50.851856 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:51.577781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.577748 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2"] Oct 30 09:15:51.578475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578454 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:51.578573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578475 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b220a92-0d46-4e9d-8341-33d67bb05173" containerName="cleanup" Oct 30 09:15:51.578573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578508 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:51.578573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578516 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:51.578573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578529 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:51.578573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578537 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370b934-2bba-435c-9db6-ed4655c88448" containerName="cleanup" Oct 30 09:15:51.578821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.578685 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef976272-a7b6-4c88-bf10-6d69cfd8afd7" containerName="cleanup" Oct 30 09:15:51.582140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.582110 4920 generic.go:356] "Generic (PLEG): container finished" podID="7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" containerID="bcf2f564fb6decbc6a7e6bc07eed5de0b991ecee78bd813f7bfa7b80ad2341af" exitCode=0 Oct 30 09:15:51.582276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.582155 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.582276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.582233 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" event={"ID":"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2","Type":"ContainerDied","Data":"bcf2f564fb6decbc6a7e6bc07eed5de0b991ecee78bd813f7bfa7b80ad2341af"} Oct 30 09:15:51.610880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.610838 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2"] Oct 30 09:15:51.728493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.728453 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g625\" (UniqueName: \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.728661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.728571 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.728783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.728763 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.829728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.829639 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.829728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.829689 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g625\" (UniqueName: \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.829977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.829740 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.830477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.830452 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.832040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.832010 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.848292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.848266 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g625\" (UniqueName: \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\") pod \"maas-api-revoke-sub-d3cc9eeb-xvmm2\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:51.922189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:51.922144 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:52.222358 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:52.222323 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda518c097_78d5_4887_b2e2_b10a34bd4e7a.slice/crio-6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b WatchSource:0}: Error finding container 6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b: Status 404 returned error can't find the container with id 6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b Oct 30 09:15:52.223032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.222970 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2"] Oct 30 09:15:52.497890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.497793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:52.590837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.590796 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" event={"ID":"a518c097-78d5-4887-b2e2-b10a34bd4e7a","Type":"ContainerStarted","Data":"bd994a7b600bad8bafbe5cac8add36f1cc3613892294ac190fee5afc03df4e7c"} Oct 30 09:15:52.591368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.590853 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" event={"ID":"a518c097-78d5-4887-b2e2-b10a34bd4e7a","Type":"ContainerStarted","Data":"6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b"} Oct 30 09:15:52.630649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.630566 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" podStartSLOduration=1.630547032 podStartE2EDuration="1.630547032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:52.628090633 +0000 UTC m=+1316.351517775" watchObservedRunningTime="2026-10-30 09:15:52.630547032 +0000 UTC m=+1316.353974166" Oct 30 09:15:52.748986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.748927 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:52.840575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.840544 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\") pod \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " Oct 30 09:15:52.840791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.840599 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\") pod \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " Oct 30 09:15:52.840791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.840689 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\") pod \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\" (UID: \"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2\") " Oct 30 09:15:52.841065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.841034 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca" pod "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" (UID: "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:52.842828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.842802 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca" pod "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" (UID: "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:15:52.842828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.842822 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp" pod "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" (UID: "7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2"). InnerVolumeSpecName "kube-api-access-hwdlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:52.942757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.942722 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-kube-api-access-hwdlp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:52.942757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.942752 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:52.942757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:52.942763 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:53.599081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:53.599046 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" Oct 30 09:15:53.599592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:53.599045 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6" event={"ID":"7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2","Type":"ContainerDied","Data":"063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445"} Oct 30 09:15:53.599592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:53.599185 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063e0ef868d1b53841fd874b3672cf30637d3e39b137276f0ea1dab698065445" Oct 30 09:15:53.600978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:53.600949 4920 generic.go:356] "Generic (PLEG): container finished" podID="a518c097-78d5-4887-b2e2-b10a34bd4e7a" containerID="bd994a7b600bad8bafbe5cac8add36f1cc3613892294ac190fee5afc03df4e7c" exitCode=0 Oct 30 09:15:53.601120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:53.601025 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" event={"ID":"a518c097-78d5-4887-b2e2-b10a34bd4e7a","Type":"ContainerDied","Data":"bd994a7b600bad8bafbe5cac8add36f1cc3613892294ac190fee5afc03df4e7c"} Oct 30 09:15:54.091976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.091946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:54.382261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.382155 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:15:54.382700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.382685 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" containerName="revoke-keys" Oct 30 09:15:54.382752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.382701 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" containerName="revoke-keys" Oct 30 09:15:54.382806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.382796 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" containerName="revoke-keys" Oct 30 09:15:54.386006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.385990 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.415356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.415327 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-176b6c-openshift-default-dockercfg-k67tz\"" Oct 30 09:15:54.418479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.418447 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:15:54.563312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.563260 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.563494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.563440 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.563686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.563662 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.563764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.563745 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.564055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.564024 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.564190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.564095 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngg5\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.564294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.564201 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.564294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.564282 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.564384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.564317 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.665813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665715 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.665813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665756 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.665813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665777 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.665813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665805 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngg5\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665834 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665881 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665910 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.665964 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.666016 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.666165 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.666433 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.666657 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.666889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.666839 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.667746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.667710 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.668781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.668754 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.668886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.668863 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.689105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.689048 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngg5\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.695211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.695184 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.702907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.702074 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:54.799340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.799309 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:54.973820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.973721 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\") pod \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " Oct 30 09:15:54.973972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.973829 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\") pod \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " Oct 30 09:15:54.973972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.973855 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\" (UniqueName: \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\") pod \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\" (UID: \"a518c097-78d5-4887-b2e2-b10a34bd4e7a\") " Oct 30 09:15:54.974204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.974164 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca" pod "a518c097-78d5-4887-b2e2-b10a34bd4e7a" (UID: "a518c097-78d5-4887-b2e2-b10a34bd4e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:15:54.974678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.974656 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:54.975830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.975806 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca" pod "a518c097-78d5-4887-b2e2-b10a34bd4e7a" (UID: "a518c097-78d5-4887-b2e2-b10a34bd4e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:15:54.975921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:54.975902 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625" pod "a518c097-78d5-4887-b2e2-b10a34bd4e7a" (UID: "a518c097-78d5-4887-b2e2-b10a34bd4e7a"). InnerVolumeSpecName "kube-api-access-6g625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:15:55.075643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.075605 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a518c097-78d5-4887-b2e2-b10a34bd4e7a-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:55.075643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.075638 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g625\" (UniqueName: \"kubernetes.io/projected/a518c097-78d5-4887-b2e2-b10a34bd4e7a-kube-api-access-6g625\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:15:55.097982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.097950 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:15:55.106271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.106206 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:55.106355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.106313 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:55.106405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.106387 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:55.632267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.632201 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" event={"ID":"4e969a6e-ab0f-49c0-bd19-9c8670209bdc","Type":"ContainerStarted","Data":"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e"} Oct 30 09:15:55.632480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.632285 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" event={"ID":"4e969a6e-ab0f-49c0-bd19-9c8670209bdc","Type":"ContainerStarted","Data":"2b85cf5321716ee9fa62eac7658ae831d49b9e03b6a085c70741080ca2bcaa82"} Oct 30 09:15:55.633937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.633920 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" Oct 30 09:15:55.634026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.633967 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2" event={"ID":"a518c097-78d5-4887-b2e2-b10a34bd4e7a","Type":"ContainerDied","Data":"6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b"} Oct 30 09:15:55.634026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.634017 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5b45e7501ea69d28fce3a9a36e0c16d8cb2f2a02d402bc7c423638941c5a5b" Oct 30 09:15:55.674177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.674125 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podStartSLOduration=1.674110594 podStartE2EDuration="1.674110594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:55.672848079 +0000 UTC m=+1319.396275221" watchObservedRunningTime="2026-10-30 09:15:55.674110594 +0000 UTC m=+1319.397537737" Oct 30 09:15:55.712358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.712307 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:15:55.788612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:55.788580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:56.714256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.714201 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 09:15:56.714806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.714285 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 09:15:56.952732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.952649 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2"] Oct 30 09:15:56.953182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.953168 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a518c097-78d5-4887-b2e2-b10a34bd4e7a" containerName="revoke-keys" Oct 30 09:15:56.953242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.953184 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a518c097-78d5-4887-b2e2-b10a34bd4e7a" containerName="revoke-keys" Oct 30 09:15:56.953323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.953312 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a518c097-78d5-4887-b2e2-b10a34bd4e7a" containerName="revoke-keys" Oct 30 09:15:56.957771 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.957749 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:56.974670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.974610 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2"] Oct 30 09:15:56.981832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.981802 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:15:56.981979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.981879 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:15:56.986865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.986838 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:15:56.986968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.986884 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:15:56.995667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.995643 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7hd\" (UniqueName: \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:56.995807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.995711 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:56.995852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:56.995813 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.097120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.097082 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7hd\" (UniqueName: \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.097336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.097186 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.097336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.097283 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.129013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.127848 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.136927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.136893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.138602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.138579 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7hd\" (UniqueName: \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\") pod \"maas-api-revoke-sub-ff1106c7-554j2\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.271841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.271753 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:15:57.440369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.440315 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:57.441763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.440970 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk"] Oct 30 09:15:57.454641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.453912 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk"] Oct 30 09:15:57.454641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.454041 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.538423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.538386 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.538702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.538687 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.538864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.538853 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmx8\" (UniqueName: \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.622240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.619760 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2"] Oct 30 09:15:57.630700 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:57.630664 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fcba3c_543f_4ad9_99fb_43fd7ef88057.slice/crio-b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c WatchSource:0}: Error finding container b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c: Status 404 returned error can't find the container with id b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c Oct 30 09:15:57.639958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.639928 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmx8\" (UniqueName: \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.640255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.640212 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.640347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.640287 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.640942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.640919 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.642840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.642815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.650038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.650013 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" event={"ID":"90fcba3c-543f-4ad9-99fb-43fd7ef88057","Type":"ContainerStarted","Data":"b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c"} Oct 30 09:15:57.657931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.657911 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmx8\" (UniqueName: \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\") pod \"maas-api-revoke-sub-987b2411-8b2tk\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.708242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.706705 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 09:15:57.708242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.706766 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 09:15:57.769554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.769522 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:15:57.933164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.933123 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj"] Oct 30 09:15:57.940141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.940110 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:57.952159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:57.951927 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj"] Oct 30 09:15:58.045024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.044990 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wjb\" (UniqueName: \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.045024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.045024 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.045182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.045043 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.130662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.130196 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk"] Oct 30 09:15:58.130662 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:58.130358 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda5aa14_b10a_48c3_8b2b_5e61752471e1.slice/crio-c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243 WatchSource:0}: Error finding container c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243: Status 404 returned error can't find the container with id c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243 Oct 30 09:15:58.147003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.146974 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wjb\" (UniqueName: \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.147070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.147024 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.147070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.147053 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.147898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.147874 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.149992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.149967 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.165592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.165561 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wjb\" (UniqueName: \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\") pod \"maas-api-revoke-sub-645924b4-bshpj\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.258033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.257997 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:15:58.596273 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:15:58.596240 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd93536_8df5_4d75_ba29_0b46b169d5a1.slice/crio-29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28 WatchSource:0}: Error finding container 29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28: Status 404 returned error can't find the container with id 29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28 Oct 30 09:15:58.607237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.597610 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj"] Oct 30 09:15:58.657417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.657376 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" event={"ID":"90fcba3c-543f-4ad9-99fb-43fd7ef88057","Type":"ContainerStarted","Data":"f75706ea8ab0e38e95bcf4e55e018a9aebe2296bd2597dbfd1a4901ac1c22626"} Oct 30 09:15:58.658759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.658732 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" event={"ID":"edd93536-8df5-4d75-ba29-0b46b169d5a1","Type":"ContainerStarted","Data":"29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28"} Oct 30 09:15:58.660172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.660147 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" event={"ID":"cda5aa14-b10a-48c3-8b2b-5e61752471e1","Type":"ContainerStarted","Data":"fbaf2d39532cf54f10827e0059f437ecc3179590f402d80332f3f801bd8312ce"} Oct 30 09:15:58.660283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.660189 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" event={"ID":"cda5aa14-b10a-48c3-8b2b-5e61752471e1","Type":"ContainerStarted","Data":"c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243"} Oct 30 09:15:58.694957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.694903 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" podStartSLOduration=2.694889038 podStartE2EDuration="2.694889038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:58.692953332 +0000 UTC m=+1322.416380478" watchObservedRunningTime="2026-10-30 09:15:58.694889038 +0000 UTC m=+1322.418316180" Oct 30 09:15:58.726249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.724377 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 09:15:58.726249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.724436 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 09:15:58.748479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.748408 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" podStartSLOduration=1.748388395 podStartE2EDuration="1.748388395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:58.743637297 +0000 UTC m=+1322.467064440" watchObservedRunningTime="2026-10-30 09:15:58.748388395 +0000 UTC m=+1322.471815539" Oct 30 09:15:58.972902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:58.972868 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:15:59.668626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.668590 4920 generic.go:356] "Generic (PLEG): container finished" podID="cda5aa14-b10a-48c3-8b2b-5e61752471e1" containerID="fbaf2d39532cf54f10827e0059f437ecc3179590f402d80332f3f801bd8312ce" exitCode=0 Oct 30 09:15:59.668829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.668686 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" event={"ID":"cda5aa14-b10a-48c3-8b2b-5e61752471e1","Type":"ContainerDied","Data":"fbaf2d39532cf54f10827e0059f437ecc3179590f402d80332f3f801bd8312ce"} Oct 30 09:15:59.670163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.670144 4920 generic.go:356] "Generic (PLEG): container finished" podID="90fcba3c-543f-4ad9-99fb-43fd7ef88057" containerID="f75706ea8ab0e38e95bcf4e55e018a9aebe2296bd2597dbfd1a4901ac1c22626" exitCode=0 Oct 30 09:15:59.670277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.670232 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" event={"ID":"90fcba3c-543f-4ad9-99fb-43fd7ef88057","Type":"ContainerDied","Data":"f75706ea8ab0e38e95bcf4e55e018a9aebe2296bd2597dbfd1a4901ac1c22626"} Oct 30 09:15:59.671806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.671783 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" event={"ID":"edd93536-8df5-4d75-ba29-0b46b169d5a1","Type":"ContainerStarted","Data":"e4501497189ce9ea427731324ba020725086c4eb2effef78cca3187e89cdd619"} Oct 30 09:15:59.707241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.705607 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 09:15:59.707241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.705667 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 09:15:59.789758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:15:59.789710 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" podStartSLOduration=2.789696674 podStartE2EDuration="2.789696674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:59.789174911 +0000 UTC m=+1323.512602054" watchObservedRunningTime="2026-10-30 09:15:59.789696674 +0000 UTC m=+1323.513123816" Oct 30 09:16:00.679719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.679687 4920 generic.go:356] "Generic (PLEG): container finished" podID="edd93536-8df5-4d75-ba29-0b46b169d5a1" containerID="e4501497189ce9ea427731324ba020725086c4eb2effef78cca3187e89cdd619" exitCode=0 Oct 30 09:16:00.680172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.679776 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" event={"ID":"edd93536-8df5-4d75-ba29-0b46b169d5a1","Type":"ContainerDied","Data":"e4501497189ce9ea427731324ba020725086c4eb2effef78cca3187e89cdd619"} Oct 30 09:16:00.684526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.684494 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:00.728324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.727354 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 09:16:00.728324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.727419 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 09:16:00.973569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.973539 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:16:00.976953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:00.976924 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:16:01.080194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080150 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\") pod \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " Oct 30 09:16:01.080408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080240 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\") pod \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " Oct 30 09:16:01.080408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080362 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\" (UniqueName: \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\") pod \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " Oct 30 09:16:01.080408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080398 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\") pod \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " Oct 30 09:16:01.080600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080439 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\" (UniqueName: \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\") pod \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\" (UID: \"cda5aa14-b10a-48c3-8b2b-5e61752471e1\") " Oct 30 09:16:01.080600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080549 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\") pod \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\" (UID: \"90fcba3c-543f-4ad9-99fb-43fd7ef88057\") " Oct 30 09:16:01.080697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080647 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca" pod "cda5aa14-b10a-48c3-8b2b-5e61752471e1" (UID: "cda5aa14-b10a-48c3-8b2b-5e61752471e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:01.081013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.080987 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.081123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.081007 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca" pod "90fcba3c-543f-4ad9-99fb-43fd7ef88057" (UID: "90fcba3c-543f-4ad9-99fb-43fd7ef88057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:01.082925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.082898 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca" pod "cda5aa14-b10a-48c3-8b2b-5e61752471e1" (UID: "cda5aa14-b10a-48c3-8b2b-5e61752471e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:01.083337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.083311 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd" pod "90fcba3c-543f-4ad9-99fb-43fd7ef88057" (UID: "90fcba3c-543f-4ad9-99fb-43fd7ef88057"). InnerVolumeSpecName "kube-api-access-kd7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:01.084044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.084008 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca" pod "90fcba3c-543f-4ad9-99fb-43fd7ef88057" (UID: "90fcba3c-543f-4ad9-99fb-43fd7ef88057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:01.084044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.084025 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8" pod "cda5aa14-b10a-48c3-8b2b-5e61752471e1" (UID: "cda5aa14-b10a-48c3-8b2b-5e61752471e1"). InnerVolumeSpecName "kube-api-access-dpmx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:01.135802 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:16:01.135752 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:16:01.182777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.182740 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda5aa14-b10a-48c3-8b2b-5e61752471e1-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.182777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.182780 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7hd\" (UniqueName: \"kubernetes.io/projected/90fcba3c-543f-4ad9-99fb-43fd7ef88057-kube-api-access-kd7hd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.182974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.182797 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.182974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.182814 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmx8\" (UniqueName: \"kubernetes.io/projected/cda5aa14-b10a-48c3-8b2b-5e61752471e1-kube-api-access-dpmx8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.182974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.182829 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fcba3c-543f-4ad9-99fb-43fd7ef88057-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:01.690659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.690618 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" Oct 30 09:16:01.691157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.690614 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk" event={"ID":"cda5aa14-b10a-48c3-8b2b-5e61752471e1","Type":"ContainerDied","Data":"c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243"} Oct 30 09:16:01.691157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.690785 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62e27d5a9f655cc99d134e8fd135fef560c0df59d52e013d9f4c12d94756243" Oct 30 09:16:01.692476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.692445 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" Oct 30 09:16:01.692588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.692492 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2" event={"ID":"90fcba3c-543f-4ad9-99fb-43fd7ef88057","Type":"ContainerDied","Data":"b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c"} Oct 30 09:16:01.692588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.692537 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77fa7281e32c73558644f09356d2631241a38b8356454c310f351ce4875c81c" Oct 30 09:16:01.901000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.900959 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:16:01.991713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.991612 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\") pod \"edd93536-8df5-4d75-ba29-0b46b169d5a1\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " Oct 30 09:16:01.991713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.991692 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\" (UniqueName: \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\") pod \"edd93536-8df5-4d75-ba29-0b46b169d5a1\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " Oct 30 09:16:01.991945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.991788 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\") pod \"edd93536-8df5-4d75-ba29-0b46b169d5a1\" (UID: \"edd93536-8df5-4d75-ba29-0b46b169d5a1\") " Oct 30 09:16:01.992149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.992122 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca" pod "edd93536-8df5-4d75-ba29-0b46b169d5a1" (UID: "edd93536-8df5-4d75-ba29-0b46b169d5a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:01.993936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.993909 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca" pod "edd93536-8df5-4d75-ba29-0b46b169d5a1" (UID: "edd93536-8df5-4d75-ba29-0b46b169d5a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:01.994029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:01.993939 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb" pod "edd93536-8df5-4d75-ba29-0b46b169d5a1" (UID: "edd93536-8df5-4d75-ba29-0b46b169d5a1"). InnerVolumeSpecName "kube-api-access-68wjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:02.093483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.093437 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wjb\" (UniqueName: \"kubernetes.io/projected/edd93536-8df5-4d75-ba29-0b46b169d5a1-kube-api-access-68wjb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:02.093483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.093483 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:02.093684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.093500 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd93536-8df5-4d75-ba29-0b46b169d5a1-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:02.459233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.459188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:02.635038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.635001 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:16:02.635480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.635453 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:16:02.636797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.636777 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:16:02.702344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.702265 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" Oct 30 09:16:02.702814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.702340 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj" event={"ID":"edd93536-8df5-4d75-ba29-0b46b169d5a1","Type":"ContainerDied","Data":"29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28"} Oct 30 09:16:02.702814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:02.702388 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e3a7367da5d7ac72bd3b0a0a884beafcd3591d15ed46d7a6c762644075bd28" Oct 30 09:16:04.124373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:04.124331 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:05.923535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:05.923503 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:07.669668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:07.669626 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:09.464642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:09.464605 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:11.063560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.063520 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:11.443879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.443785 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp"] Oct 30 09:16:11.444344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444323 4920 cpu_manager.go:414] "containerMap: removing container" podUID="edd93536-8df5-4d75-ba29-0b46b169d5a1" containerName="revoke-keys" Oct 30 09:16:11.444344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444341 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd93536-8df5-4d75-ba29-0b46b169d5a1" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444375 4920 cpu_manager.go:414] "containerMap: removing container" podUID="cda5aa14-b10a-48c3-8b2b-5e61752471e1" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444381 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda5aa14-b10a-48c3-8b2b-5e61752471e1" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444397 4920 cpu_manager.go:414] "containerMap: removing container" podUID="90fcba3c-543f-4ad9-99fb-43fd7ef88057" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444402 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fcba3c-543f-4ad9-99fb-43fd7ef88057" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444475 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda5aa14-b10a-48c3-8b2b-5e61752471e1" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444485 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fcba3c-543f-4ad9-99fb-43fd7ef88057" containerName="revoke-keys" Oct 30 09:16:11.444511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.444491 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd93536-8df5-4d75-ba29-0b46b169d5a1" containerName="revoke-keys" Oct 30 09:16:11.448407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.448387 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.463504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.463476 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:16:11.475118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.475067 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:16:11.475321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.475133 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:16:11.475321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.475168 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:16:11.478281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.478259 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp"] Oct 30 09:16:11.503489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.503452 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.503656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.503493 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.503712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.503659 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52qv\" (UniqueName: \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.604517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.604483 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.604517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.604520 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.604782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.604625 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52qv\" (UniqueName: \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.605407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.605376 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.606898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.606874 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.623660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.623634 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52qv\" (UniqueName: \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\") pod \"maas-api-revoke-sub-e48dc1c8-j72tp\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:11.762553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:11.762466 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:12.067563 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:16:12.067525 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd700e465_3d43_40c2_baf4_f740a4cd37b2.slice/crio-7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519 WatchSource:0}: Error finding container 7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519: Status 404 returned error can't find the container with id 7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519 Oct 30 09:16:12.068662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:12.068633 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp"] Oct 30 09:16:12.672853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:12.672768 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:12.788149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:12.788113 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" event={"ID":"d700e465-3d43-40c2-baf4-f740a4cd37b2","Type":"ContainerStarted","Data":"4299d2671067fa3dc7a8934981960340e7e0a545230f9d43a6bca2a48d280282"} Oct 30 09:16:12.788358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:12.788163 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" event={"ID":"d700e465-3d43-40c2-baf4-f740a4cd37b2","Type":"ContainerStarted","Data":"7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519"} Oct 30 09:16:12.839618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:12.839559 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" podStartSLOduration=1.839543521 podStartE2EDuration="1.839543521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:12.826568475 +0000 UTC m=+1336.549995619" watchObservedRunningTime="2026-10-30 09:16:12.839543521 +0000 UTC m=+1336.562970663" Oct 30 09:16:13.796205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:13.796169 4920 generic.go:356] "Generic (PLEG): container finished" podID="d700e465-3d43-40c2-baf4-f740a4cd37b2" containerID="4299d2671067fa3dc7a8934981960340e7e0a545230f9d43a6bca2a48d280282" exitCode=0 Oct 30 09:16:13.796631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:13.796233 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" event={"ID":"d700e465-3d43-40c2-baf4-f740a4cd37b2","Type":"ContainerDied","Data":"4299d2671067fa3dc7a8934981960340e7e0a545230f9d43a6bca2a48d280282"} Oct 30 09:16:14.160474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:14.160392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:15.038500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.038113 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:15.152296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.151616 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\") pod \"d700e465-3d43-40c2-baf4-f740a4cd37b2\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " Oct 30 09:16:15.152296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.151778 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\") pod \"d700e465-3d43-40c2-baf4-f740a4cd37b2\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " Oct 30 09:16:15.152296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.151861 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\" (UniqueName: \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\") pod \"d700e465-3d43-40c2-baf4-f740a4cd37b2\" (UID: \"d700e465-3d43-40c2-baf4-f740a4cd37b2\") " Oct 30 09:16:15.173243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.172899 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca" pod "d700e465-3d43-40c2-baf4-f740a4cd37b2" (UID: "d700e465-3d43-40c2-baf4-f740a4cd37b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:15.180848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.180765 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv" pod "d700e465-3d43-40c2-baf4-f740a4cd37b2" (UID: "d700e465-3d43-40c2-baf4-f740a4cd37b2"). InnerVolumeSpecName "kube-api-access-j52qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:15.222459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.222366 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca" pod "d700e465-3d43-40c2-baf4-f740a4cd37b2" (UID: "d700e465-3d43-40c2-baf4-f740a4cd37b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:15.256085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.255993 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:15.256085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.256039 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d700e465-3d43-40c2-baf4-f740a4cd37b2-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:15.256085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.256055 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52qv\" (UniqueName: \"kubernetes.io/projected/d700e465-3d43-40c2-baf4-f740a4cd37b2-kube-api-access-j52qv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:15.676056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.676022 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:15.802360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.802324 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2"] Oct 30 09:16:15.802863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.802847 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d700e465-3d43-40c2-baf4-f740a4cd37b2" containerName="revoke-keys" Oct 30 09:16:15.802863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.802863 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d700e465-3d43-40c2-baf4-f740a4cd37b2" containerName="revoke-keys" Oct 30 09:16:15.803011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.802934 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d700e465-3d43-40c2-baf4-f740a4cd37b2" containerName="revoke-keys" Oct 30 09:16:15.806194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.806167 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:15.814850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.814826 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" Oct 30 09:16:15.814987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.814874 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp" event={"ID":"d700e465-3d43-40c2-baf4-f740a4cd37b2","Type":"ContainerDied","Data":"7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519"} Oct 30 09:16:15.814987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.814929 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d8a58d02a426f88c8adc3a048880b15cf149a5beb5af4dd9f24fc3139ff5519" Oct 30 09:16:15.824640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.824612 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2"] Oct 30 09:16:15.966515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.966435 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvt52\" (UniqueName: \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:15.966515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.966478 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:15.966714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:15.966645 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.068009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.067975 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.068483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.068039 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvt52\" (UniqueName: \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.068483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.068068 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.068671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.068651 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.070375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.070355 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.086364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.086337 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvt52\" (UniqueName: \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\") pod \"maas-api-revoke-sub-b148c14b-s2sb2\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.121854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.121827 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:16.419547 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:16:16.419507 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1947729_466b_486c_a88f_65c71ef39fbb.slice/crio-e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060 WatchSource:0}: Error finding container e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060: Status 404 returned error can't find the container with id e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060 Oct 30 09:16:16.422255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.422186 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2"] Oct 30 09:16:16.823886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.823850 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" event={"ID":"d1947729-466b-486c-a88f-65c71ef39fbb","Type":"ContainerStarted","Data":"64ff0148c6bcabd29ee2580ecdf1499380fe888b7006961a178999edfdd91553"} Oct 30 09:16:16.824048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.823941 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" event={"ID":"d1947729-466b-486c-a88f-65c71ef39fbb","Type":"ContainerStarted","Data":"e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060"} Oct 30 09:16:16.862936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:16.862881 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" podStartSLOduration=1.862868076 podStartE2EDuration="1.862868076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:16.860097843 +0000 UTC m=+1340.583524986" watchObservedRunningTime="2026-10-30 09:16:16.862868076 +0000 UTC m=+1340.586295219" Oct 30 09:16:17.261466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:17.261432 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:17.832495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:17.832456 4920 generic.go:356] "Generic (PLEG): container finished" podID="d1947729-466b-486c-a88f-65c71ef39fbb" containerID="64ff0148c6bcabd29ee2580ecdf1499380fe888b7006961a178999edfdd91553" exitCode=0 Oct 30 09:16:17.832682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:17.832541 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" event={"ID":"d1947729-466b-486c-a88f-65c71ef39fbb","Type":"ContainerDied","Data":"64ff0148c6bcabd29ee2580ecdf1499380fe888b7006961a178999edfdd91553"} Oct 30 09:16:18.880142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:18.880101 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:19.014049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.014023 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:19.104623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.104547 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\" (UniqueName: \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\") pod \"d1947729-466b-486c-a88f-65c71ef39fbb\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " Oct 30 09:16:19.104623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.104588 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\") pod \"d1947729-466b-486c-a88f-65c71ef39fbb\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " Oct 30 09:16:19.104799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.104677 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\") pod \"d1947729-466b-486c-a88f-65c71ef39fbb\" (UID: \"d1947729-466b-486c-a88f-65c71ef39fbb\") " Oct 30 09:16:19.104970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.104946 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca" pod "d1947729-466b-486c-a88f-65c71ef39fbb" (UID: "d1947729-466b-486c-a88f-65c71ef39fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:19.106733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.106710 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca" pod "d1947729-466b-486c-a88f-65c71ef39fbb" (UID: "d1947729-466b-486c-a88f-65c71ef39fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:19.106831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.106805 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52" pod "d1947729-466b-486c-a88f-65c71ef39fbb" (UID: "d1947729-466b-486c-a88f-65c71ef39fbb"). InnerVolumeSpecName "kube-api-access-cvt52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:19.206582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.206545 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:19.206582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.206578 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvt52\" (UniqueName: \"kubernetes.io/projected/d1947729-466b-486c-a88f-65c71ef39fbb-kube-api-access-cvt52\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:19.206582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.206588 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1947729-466b-486c-a88f-65c71ef39fbb-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:19.847719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.847680 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" Oct 30 09:16:19.847719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.847693 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2" event={"ID":"d1947729-466b-486c-a88f-65c71ef39fbb","Type":"ContainerDied","Data":"e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060"} Oct 30 09:16:19.847933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:19.847736 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b6355833fa8ae2b38b05788e9a64c79568c85d589b681c358c4b39ddbfe060" Oct 30 09:16:20.449144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:20.449097 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:20.548801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:20.548763 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:22.179008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:22.178968 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:23.871056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:23.871019 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:24.552853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:24.552817 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:16:25.536715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:25.536641 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:26.160355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.160321 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr"] Oct 30 09:16:26.160849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.160835 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d1947729-466b-486c-a88f-65c71ef39fbb" containerName="revoke-keys" Oct 30 09:16:26.160890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.160851 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1947729-466b-486c-a88f-65c71ef39fbb" containerName="revoke-keys" Oct 30 09:16:26.160981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.160972 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1947729-466b-486c-a88f-65c71ef39fbb" containerName="revoke-keys" Oct 30 09:16:26.164526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.164511 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.181067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.181040 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.181188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.181097 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg25r\" (UniqueName: \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.181249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.181203 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.183179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.183155 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr"] Oct 30 09:16:26.190846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.190819 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:16:26.190846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.190837 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:16:26.190846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.190819 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:16:26.191103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.190881 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:16:26.281830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.281796 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.282016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.281850 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg25r\" (UniqueName: \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.282103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.282025 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.282652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.282627 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.284309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.284285 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.302347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.302317 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg25r\" (UniqueName: \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\") pod \"maas-api-revoke-sub-3f2c8dba-tbcvr\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.478313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.478201 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:26.780913 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:16:26.780877 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded18c35b_2c79_4a9f_b0e8_ede7198ce194.slice/crio-f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962 WatchSource:0}: Error finding container f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962: Status 404 returned error can't find the container with id f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962 Oct 30 09:16:26.782353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.782327 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr"] Oct 30 09:16:26.938603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:26.938555 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" event={"ID":"ed18c35b-2c79-4a9f-b0e8-ede7198ce194","Type":"ContainerStarted","Data":"f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962"} Oct 30 09:16:27.317706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:27.317657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:27.946714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:27.946673 4920 generic.go:356] "Generic (PLEG): container finished" podID="ed18c35b-2c79-4a9f-b0e8-ede7198ce194" containerID="9956a2e5894b9f9acdab5b9546899a3cde61387ba9fa8de2195189d642cb2837" exitCode=0 Oct 30 09:16:27.947280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:27.946754 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" event={"ID":"ed18c35b-2c79-4a9f-b0e8-ede7198ce194","Type":"ContainerDied","Data":"9956a2e5894b9f9acdab5b9546899a3cde61387ba9fa8de2195189d642cb2837"} Oct 30 09:16:28.955637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:28.955595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:29.138125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.138085 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:29.212081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.211975 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\") pod \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " Oct 30 09:16:29.212081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.212059 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\") pod \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " Oct 30 09:16:29.212395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.212137 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\" (UniqueName: \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\") pod \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\" (UID: \"ed18c35b-2c79-4a9f-b0e8-ede7198ce194\") " Oct 30 09:16:29.212464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.212410 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca" pod "ed18c35b-2c79-4a9f-b0e8-ede7198ce194" (UID: "ed18c35b-2c79-4a9f-b0e8-ede7198ce194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:16:29.212886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.212574 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:29.215133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.215104 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r" pod "ed18c35b-2c79-4a9f-b0e8-ede7198ce194" (UID: "ed18c35b-2c79-4a9f-b0e8-ede7198ce194"). InnerVolumeSpecName "kube-api-access-sg25r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:16:29.217562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.217533 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca" pod "ed18c35b-2c79-4a9f-b0e8-ede7198ce194" (UID: "ed18c35b-2c79-4a9f-b0e8-ede7198ce194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:16:29.313599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.313562 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:29.313599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.313595 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg25r\" (UniqueName: \"kubernetes.io/projected/ed18c35b-2c79-4a9f-b0e8-ede7198ce194-kube-api-access-sg25r\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:16:29.961254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.961198 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" event={"ID":"ed18c35b-2c79-4a9f-b0e8-ede7198ce194","Type":"ContainerDied","Data":"f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962"} Oct 30 09:16:29.961254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.961249 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr" Oct 30 09:16:29.961795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:29.961279 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7bdde6eeed071088a484948e80971f670e0163e51751af2cc482f3b0a9c5962" Oct 30 09:16:30.451490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:30.451451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:31.134928 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:16:31.134882 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:16:32.146333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:32.146296 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:33.677056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:33.677013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:33.776639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:33.776600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:35.762037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:35.762006 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:37.570961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:37.570923 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:39.128078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:39.128039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:39.864378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:39.864344 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:39.964497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:39.964462 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:40.654729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:40.654695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:42.141062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:42.141026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:42.575434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:42.575395 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:16:44.017477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.017442 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:44.799625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.799589 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:16:44.800156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.800136 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ed18c35b-2c79-4a9f-b0e8-ede7198ce194" containerName="revoke-keys" Oct 30 09:16:44.800291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.800160 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed18c35b-2c79-4a9f-b0e8-ede7198ce194" containerName="revoke-keys" Oct 30 09:16:44.800384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.800370 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed18c35b-2c79-4a9f-b0e8-ede7198ce194" containerName="revoke-keys" Oct 30 09:16:44.811855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.811825 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.821719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.821367 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:16:44.829626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.829601 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"distinct-w1-c5447d-kserve-self-signed-certs\"" Oct 30 09:16:44.836394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.836366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:44.893145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893109 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.893346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893154 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.893346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893294 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.893346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893321 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wmd\" (UniqueName: \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.893518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893478 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:44.893577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:44.893518 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.011999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.011955 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.012205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.012023 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.012205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.012130 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.012205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.012172 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wmd\" (UniqueName: \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.012438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.012287 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.012438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.012317 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.013379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.013349 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.013773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.013747 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.013943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.013915 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.019600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.018268 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.019600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.019162 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.032264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.032211 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wmd\" (UniqueName: \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\") pod \"distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.127712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.127618 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:45.426397 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:16:45.426365 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4878df82_03f8_4d95_8173_9cb84d7a8c8c.slice/crio-80fdd7b76181a76c6e3200a2d55845543b6a459db1e109dbc80888829687195e WatchSource:0}: Error finding container 80fdd7b76181a76c6e3200a2d55845543b6a459db1e109dbc80888829687195e: Status 404 returned error can't find the container with id 80fdd7b76181a76c6e3200a2d55845543b6a459db1e109dbc80888829687195e Oct 30 09:16:45.429689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.429658 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:16:45.756850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:45.756747 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:46.079473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:46.079434 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" event={"ID":"4878df82-03f8-4d95-8173-9cb84d7a8c8c","Type":"ContainerStarted","Data":"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb"} Oct 30 09:16:46.079473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:46.079480 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" event={"ID":"4878df82-03f8-4d95-8173-9cb84d7a8c8c","Type":"ContainerStarted","Data":"80fdd7b76181a76c6e3200a2d55845543b6a459db1e109dbc80888829687195e"} Oct 30 09:16:46.117525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:46.117470 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podStartSLOduration=2.117455381 podStartE2EDuration="2.117455381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:46.116664534 +0000 UTC m=+1369.840091668" watchObservedRunningTime="2026-10-30 09:16:46.117455381 +0000 UTC m=+1369.840882523" Oct 30 09:16:46.127898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:46.127865 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:47.474613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:47.474582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:48.236700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:48.236661 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:48.969040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:48.969004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:50.566653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:50.566610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:52.110908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:52.110874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:53.632470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:53.632433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:55.128000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:55.127962 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:55.141111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:55.141082 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:55.158086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:55.158054 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:16:55.252148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:55.252100 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:56.383087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:56.383045 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:16:56.826710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:56.826669 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:56.950715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:56.950683 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:16:58.745463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:16:58.745429 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:00.357677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:00.357607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:01.136317 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:17:01.136276 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:17:01.963452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:01.963417 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:03.572036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:03.572000 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:05.235578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:05.235509 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:06.695500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.695464 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:17:06.700851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.700825 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.717083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.717051 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"openshift-service-ca.crt\"" Oct 30 09:17:06.728537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.728504 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"kube-root-ca.crt\"" Oct 30 09:17:06.728770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.728519 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"default-dockercfg-tfdgb\"" Oct 30 09:17:06.728770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.728567 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3a61e1\"/\"distinct-2-w4-3a61e1-kserve-self-signed-certs\"" Oct 30 09:17:06.734286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.734258 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:17:06.737772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.737729 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:06.757868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.757838 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.758030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.758005 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.758077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.758045 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.758112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.758087 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.758112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.758102 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.758210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.758129 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7rd\" (UniqueName: \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.857961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.857878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:06.859077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859055 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859087 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859137 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859160 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859185 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7rd\" (UniqueName: \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859247 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859534 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859549 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.859630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.859619 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.861496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.861469 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.861917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.861895 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:06.878935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:06.878898 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7rd\" (UniqueName: \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\") pod \"distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:07.015473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:07.015375 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:07.321058 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:17:07.321025 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90ecec18_5a58_4ed0_8831_aa0f3a23d51f.slice/crio-1e5594f9e258b4afab8eb14d6a71f4868edf8bd4096e85d86770c7de5d1e1dfb WatchSource:0}: Error finding container 1e5594f9e258b4afab8eb14d6a71f4868edf8bd4096e85d86770c7de5d1e1dfb: Status 404 returned error can't find the container with id 1e5594f9e258b4afab8eb14d6a71f4868edf8bd4096e85d86770c7de5d1e1dfb Oct 30 09:17:07.322735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:07.322711 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:17:08.246637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:08.246596 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" event={"ID":"90ecec18-5a58-4ed0-8831-aa0f3a23d51f","Type":"ContainerStarted","Data":"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca"} Oct 30 09:17:08.246637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:08.246641 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" event={"ID":"90ecec18-5a58-4ed0-8831-aa0f3a23d51f","Type":"ContainerStarted","Data":"1e5594f9e258b4afab8eb14d6a71f4868edf8bd4096e85d86770c7de5d1e1dfb"} Oct 30 09:17:08.286078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:08.286022 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podStartSLOduration=2.286009033 podStartE2EDuration="2.286009033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:08.283776889 +0000 UTC m=+1392.007204031" watchObservedRunningTime="2026-10-30 09:17:08.286009033 +0000 UTC m=+1392.009436227" Oct 30 09:17:08.569984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:08.569946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:09.015542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:09.015454 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:10.174003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:10.173956 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:12.607012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:12.606967 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:17:13.459893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:13.459853 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:13.560425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:13.560383 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:14.767680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:14.767644 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:16.234054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:16.234016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:17.016158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:17.016112 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:17.029321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:17.029287 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:17.332591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:17.332555 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:17:17.745167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:17.745084 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:19.252136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.252102 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:19.551423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.551387 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 09:17:19.556260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.556236 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.574202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.574174 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 09:17:19.582610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.582564 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"kube-root-ca.crt\"" Oct 30 09:17:19.582610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.582568 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"openshift-service-ca.crt\"" Oct 30 09:17:19.582610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.582618 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"distinct-w0-2f69c8-kserve-self-signed-certs\"" Oct 30 09:17:19.583867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.583846 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"default-dockercfg-lh8pz\"" Oct 30 09:17:19.596697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.596664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:19.709181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709146 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2p2\" (UniqueName: \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.709181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709185 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.709495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709251 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.709495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709397 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.709495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709433 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.709495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.709456 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810282 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810331 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810357 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810403 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2p2\" (UniqueName: \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810421 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810460 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810808 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810845 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.810937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.810898 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.812700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.812674 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.812782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.812750 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.829567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.829537 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2p2\" (UniqueName: \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\") pod \"distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:19.873387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:19.873349 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:20.181246 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:17:20.180710 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25e769a8_fd84_47f3_b5e1_b44328f557bd.slice/crio-1d86fec4456c7d3cf93d2265aaa6c38b019387988a31e78f5aeb5d1a437e7cc2 WatchSource:0}: Error finding container 1d86fec4456c7d3cf93d2265aaa6c38b019387988a31e78f5aeb5d1a437e7cc2: Status 404 returned error can't find the container with id 1d86fec4456c7d3cf93d2265aaa6c38b019387988a31e78f5aeb5d1a437e7cc2 Oct 30 09:17:20.181246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.181194 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 09:17:20.350279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.350239 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" event={"ID":"25e769a8-fd84-47f3-b5e1-b44328f557bd","Type":"ContainerStarted","Data":"d31dbe5ed19a87aeb019368acd547d2878b6e17fac88207f7c50e2d55e959833"} Oct 30 09:17:20.350279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.350285 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" event={"ID":"25e769a8-fd84-47f3-b5e1-b44328f557bd","Type":"ContainerStarted","Data":"1d86fec4456c7d3cf93d2265aaa6c38b019387988a31e78f5aeb5d1a437e7cc2"} Oct 30 09:17:20.390228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.390167 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podStartSLOduration=1.390153864 podStartE2EDuration="1.390153864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:20.386717919 +0000 UTC m=+1404.110145062" watchObservedRunningTime="2026-10-30 09:17:20.390153864 +0000 UTC m=+1404.113581006" Oct 30 09:17:20.861874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.861832 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:20.873761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:20.873731 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:22.457281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:22.457247 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:23.947441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:23.947398 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:24.817070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:24.817013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:25.691918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:25.691880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:26.213390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:26.213352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:26.938016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:26.937983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:27.443303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:27.443254 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:29.058470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:29.058426 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:29.874431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:29.874399 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:29.887429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:29.887396 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:30.444730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:30.444699 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:17:30.560760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:30.560721 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:31.135034 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:17:31.134997 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:17:32.165419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:32.165379 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:33.717812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.717776 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz"] Oct 30 09:17:33.721893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.721870 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.742085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.742051 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz"] Oct 30 09:17:33.749668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.749640 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:17:33.749819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.749683 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:17:33.749819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.749738 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:17:33.749819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.749747 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:17:33.766500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.766448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:33.770039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.770008 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.770290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.770268 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkxp\" (UniqueName: \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.770346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.770308 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.871106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.871073 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkxp\" (UniqueName: \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.871106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.871109 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.871400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.871146 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.871825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.871804 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.873641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.873614 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:33.889399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:33.889371 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkxp\" (UniqueName: \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\") pod \"maas-api-revoke-sub-139d508d-lhxlz\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:34.040156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:34.040121 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:34.345579 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:17:34.345546 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf13c69e_b17b_4a8b_b391_74813f8b3601.slice/crio-714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073 WatchSource:0}: Error finding container 714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073: Status 404 returned error can't find the container with id 714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073 Oct 30 09:17:34.347336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:34.347315 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz"] Oct 30 09:17:34.470859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:34.470818 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" event={"ID":"bf13c69e-b17b-4a8b-b391-74813f8b3601","Type":"ContainerStarted","Data":"714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073"} Oct 30 09:17:35.261569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:35.261535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:35.479586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:35.479508 4920 generic.go:356] "Generic (PLEG): container finished" podID="bf13c69e-b17b-4a8b-b391-74813f8b3601" containerID="55f6e77ee7c3c5b382031054fbbb4d11d745af29557459d6fd48dc81fea1b751" exitCode=0 Oct 30 09:17:35.479737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:35.479600 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" event={"ID":"bf13c69e-b17b-4a8b-b391-74813f8b3601","Type":"ContainerDied","Data":"55f6e77ee7c3c5b382031054fbbb4d11d745af29557459d6fd48dc81fea1b751"} Oct 30 09:17:36.653745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.653716 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:36.670706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.670681 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:36.696961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.696922 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\") pod \"bf13c69e-b17b-4a8b-b391-74813f8b3601\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " Oct 30 09:17:36.697090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.696985 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\") pod \"bf13c69e-b17b-4a8b-b391-74813f8b3601\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " Oct 30 09:17:36.697090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.697035 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\" (UniqueName: \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\") pod \"bf13c69e-b17b-4a8b-b391-74813f8b3601\" (UID: \"bf13c69e-b17b-4a8b-b391-74813f8b3601\") " Oct 30 09:17:36.697299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.697279 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca" pod "bf13c69e-b17b-4a8b-b391-74813f8b3601" (UID: "bf13c69e-b17b-4a8b-b391-74813f8b3601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:17:36.697449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.697426 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:36.699075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.699049 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca" pod "bf13c69e-b17b-4a8b-b391-74813f8b3601" (UID: "bf13c69e-b17b-4a8b-b391-74813f8b3601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:17:36.699175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.699156 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp" pod "bf13c69e-b17b-4a8b-b391-74813f8b3601" (UID: "bf13c69e-b17b-4a8b-b391-74813f8b3601"). InnerVolumeSpecName "kube-api-access-mfkxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:17:36.798398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.798365 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf13c69e-b17b-4a8b-b391-74813f8b3601-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:36.798398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.798402 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkxp\" (UniqueName: \"kubernetes.io/projected/bf13c69e-b17b-4a8b-b391-74813f8b3601-kube-api-access-mfkxp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:36.909298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:36.909266 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:37.493818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:37.493782 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" event={"ID":"bf13c69e-b17b-4a8b-b391-74813f8b3601","Type":"ContainerDied","Data":"714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073"} Oct 30 09:17:37.494028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:37.493846 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz" Oct 30 09:17:37.494028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:37.493855 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714af6998843836ddf03fadc0d28c5267a3a0d26489366cbe251f887f02e3073" Oct 30 09:17:37.592444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:37.592408 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:17:38.922316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:38.921971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:40.748732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:40.748695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:42.468515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:42.468465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:44.118156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:44.118117 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:45.703897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:45.703833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:47.292461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:47.292426 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:48.355012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.354973 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:48.606615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.606524 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4"] Oct 30 09:17:48.607082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.607066 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bf13c69e-b17b-4a8b-b391-74813f8b3601" containerName="revoke-keys" Oct 30 09:17:48.607082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.607082 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf13c69e-b17b-4a8b-b391-74813f8b3601" containerName="revoke-keys" Oct 30 09:17:48.607232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.607179 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf13c69e-b17b-4a8b-b391-74813f8b3601" containerName="revoke-keys" Oct 30 09:17:48.610594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.610570 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.629161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.629126 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4"] Oct 30 09:17:48.635536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.635508 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:17:48.635747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.635508 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:17:48.635856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.635555 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:17:48.636463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.636445 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:17:48.642423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.642401 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfk8\" (UniqueName: \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.642569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.642543 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.642634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.642599 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.743897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.743861 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.743897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.743904 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.744122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.743972 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfk8\" (UniqueName: \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.744836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.744803 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.746491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.746473 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.762885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.762858 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfk8\" (UniqueName: \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\") pod \"maas-api-revoke-sub-cd4c065a-8fmt4\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.925932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.925848 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:48.987212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:48.987168 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:49.236960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:49.236927 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4"] Oct 30 09:17:49.237193 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:17:49.237171 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1710a2_5f7c_4d06_8f07_4269b0d14734.slice/crio-4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58 WatchSource:0}: Error finding container 4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58: Status 404 returned error can't find the container with id 4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58 Oct 30 09:17:49.599714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:49.599675 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" event={"ID":"9d1710a2-5f7c-4d06-8f07-4269b0d14734","Type":"ContainerStarted","Data":"a9bdc2227b756d0c283a8e03cae3cf53ac7311a52b8e6c877b9b9f5a4f319be9"} Oct 30 09:17:49.600124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:49.599742 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" event={"ID":"9d1710a2-5f7c-4d06-8f07-4269b0d14734","Type":"ContainerStarted","Data":"4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58"} Oct 30 09:17:49.638891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:49.638826 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" podStartSLOduration=1.6388088280000002 podStartE2EDuration="1.638808828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:49.636297052 +0000 UTC m=+1433.359724214" watchObservedRunningTime="2026-10-30 09:17:49.638808828 +0000 UTC m=+1433.362235972" Oct 30 09:17:50.609348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:50.609312 4920 generic.go:356] "Generic (PLEG): container finished" podID="9d1710a2-5f7c-4d06-8f07-4269b0d14734" containerID="a9bdc2227b756d0c283a8e03cae3cf53ac7311a52b8e6c877b9b9f5a4f319be9" exitCode=0 Oct 30 09:17:50.609860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:50.609403 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" event={"ID":"9d1710a2-5f7c-4d06-8f07-4269b0d14734","Type":"ContainerDied","Data":"a9bdc2227b756d0c283a8e03cae3cf53ac7311a52b8e6c877b9b9f5a4f319be9"} Oct 30 09:17:50.666001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:50.665965 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:51.877710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.877645 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:51.985807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.985769 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\") pod \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " Oct 30 09:17:51.985976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.985861 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\" (UniqueName: \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\") pod \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " Oct 30 09:17:51.986035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.986014 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\") pod \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\" (UID: \"9d1710a2-5f7c-4d06-8f07-4269b0d14734\") " Oct 30 09:17:51.986310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.986282 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca" pod "9d1710a2-5f7c-4d06-8f07-4269b0d14734" (UID: "9d1710a2-5f7c-4d06-8f07-4269b0d14734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:17:51.986517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.986497 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:51.988047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.988017 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca" pod "9d1710a2-5f7c-4d06-8f07-4269b0d14734" (UID: "9d1710a2-5f7c-4d06-8f07-4269b0d14734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:17:51.988155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:51.988086 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8" pod "9d1710a2-5f7c-4d06-8f07-4269b0d14734" (UID: "9d1710a2-5f7c-4d06-8f07-4269b0d14734"). InnerVolumeSpecName "kube-api-access-xsfk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:17:52.087518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.087477 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsfk8\" (UniqueName: \"kubernetes.io/projected/9d1710a2-5f7c-4d06-8f07-4269b0d14734-kube-api-access-xsfk8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.087518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.087510 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1710a2-5f7c-4d06-8f07-4269b0d14734-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.458582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.458527 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:52.627644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.627611 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" Oct 30 09:17:52.627923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.627622 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4" event={"ID":"9d1710a2-5f7c-4d06-8f07-4269b0d14734","Type":"ContainerDied","Data":"4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58"} Oct 30 09:17:52.628030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:52.627958 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae17b48e1078a508097a20081669f64bed11d2842e1e07dae5b3f8303c45d58" Oct 30 09:17:53.201431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:53.201397 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:54.066281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.066237 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:54.563268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.563234 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:17:54.563744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.563731 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9d1710a2-5f7c-4d06-8f07-4269b0d14734" containerName="revoke-keys" Oct 30 09:17:54.563784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.563745 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1710a2-5f7c-4d06-8f07-4269b0d14734" containerName="revoke-keys" Oct 30 09:17:54.563885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.563876 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1710a2-5f7c-4d06-8f07-4269b0d14734" containerName="revoke-keys" Oct 30 09:17:54.567515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.567492 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.592631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.592594 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:17:54.617482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617446 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617503 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617627 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617662 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmf5\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617820 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617872 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617903 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.617953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.617933 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.618053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.618009 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719151 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719199 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmf5\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719294 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719335 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719359 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719384 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719410 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719519 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719562 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719802 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.719949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.719904 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.720123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.720096 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.720196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.720064 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.720277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.720209 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.722673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.722638 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.722907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.722886 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.738734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.738693 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.738912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.738866 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmf5\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:54.880445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:54.880347 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:55.212286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.212254 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:17:55.213145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.213105 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:17:55.213266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.213187 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:17:55.213332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.213297 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:17:55.550604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.550568 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:55.650707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.650661 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:55.655963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.655933 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" event={"ID":"308ae586-e0fc-463c-afbc-3f3e00773bfd","Type":"ContainerStarted","Data":"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659"} Oct 30 09:17:55.656129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.655994 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" event={"ID":"308ae586-e0fc-463c-afbc-3f3e00773bfd","Type":"ContainerStarted","Data":"cf2bf37860ba69a2ecdb37f2b00a7310b0c8568f59ad883a406aaa6e92871503"} Oct 30 09:17:55.716243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.714704 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podStartSLOduration=1.7146859490000002 podStartE2EDuration="1.714685949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:55.712328542 +0000 UTC m=+1439.435755686" watchObservedRunningTime="2026-10-30 09:17:55.714685949 +0000 UTC m=+1439.438113092" Oct 30 09:17:55.760264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.757069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:55.880609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:55.880507 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:17:56.170185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:56.170102 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:56.883437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:56.883399 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:17:56.883909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:56.883465 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:17:57.546155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:57.546114 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:57.881301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:57.881197 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:17:57.881301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:57.881270 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:17:58.880841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:58.880810 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:17:58.881335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:58.880868 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:17:59.312232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:59.312181 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:17:59.881004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:59.880963 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:17:59.881498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:17:59.881024 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:18:00.881434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:00.881402 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:18:00.881844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:00.881457 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:18:00.922242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:00.921740 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:01.147338 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:18:01.142244 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:18:01.637147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.637110 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt"] Oct 30 09:18:01.641513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.641491 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.657127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.657098 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:01.657279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.657187 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:01.670496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.670438 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:01.670496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.670439 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:01.675973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.675945 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt"] Oct 30 09:18:01.680330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.680301 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm"] Oct 30 09:18:01.685348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.685326 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.701604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.701578 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm"] Oct 30 09:18:01.722018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.721988 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.722189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.722031 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.722293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.722259 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrbq\" (UniqueName: \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.823447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823407 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrbq\" (UniqueName: \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.823447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823455 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.823713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823474 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.823713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823611 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.823713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823649 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.823867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.823763 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nlkz\" (UniqueName: \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.824614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.824585 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.826059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.826026 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.846329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.846296 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrbq\" (UniqueName: \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\") pod \"maas-api-revoke-sub-3df24201-lsnwt\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:01.880941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.880903 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 09:18:01.881117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.880961 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 09:18:01.925363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.925268 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nlkz\" (UniqueName: \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.925844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.925378 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.925844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.925407 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.926149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.926113 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.927945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.927921 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.944357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.944329 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nlkz\" (UniqueName: \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\") pod \"maas-api-revoke-sub-45edd00a-84tfm\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:01.959225 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:01.959192 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:02.010307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.010210 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:02.267292 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:02.267256 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode980f5ac_1335_4aa9_85d6_2218c3ec12b3.slice/crio-1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae WatchSource:0}: Error finding container 1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae: Status 404 returned error can't find the container with id 1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae Oct 30 09:18:02.270290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.270188 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt"] Oct 30 09:18:02.364019 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:02.363974 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d670a2_0b5e_4576_a954_4c0db0b92236.slice/crio-871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68 WatchSource:0}: Error finding container 871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68: Status 404 returned error can't find the container with id 871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68 Oct 30 09:18:02.365027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.364998 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm"] Oct 30 09:18:02.504957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.504924 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:18:02.510657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.510625 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.535164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.535111 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:18:02.649328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649295 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649374 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649435 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbh6\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649483 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649576 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649596 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649640 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649705 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.649763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.649736 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.679944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.679859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:02.715177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.715150 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" event={"ID":"e980f5ac-1335-4aa9-85d6-2218c3ec12b3","Type":"ContainerStarted","Data":"4415b407a21780fcc264ec20883bae7acf0264bf8aea6a7d2cb836943f354cab"} Oct 30 09:18:02.715337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.715201 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" event={"ID":"e980f5ac-1335-4aa9-85d6-2218c3ec12b3","Type":"ContainerStarted","Data":"1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae"} Oct 30 09:18:02.716607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.716581 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" event={"ID":"02d670a2-0b5e-4576-a954-4c0db0b92236","Type":"ContainerStarted","Data":"871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68"} Oct 30 09:18:02.750403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750367 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750408 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750454 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750472 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750488 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750561 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750624 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750653 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbh6\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.750891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.750712 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.751113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.751085 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.751434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.751405 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.751599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.751459 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.751599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.751531 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.752230 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.752190 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.753506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.753479 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.753677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.753662 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.754478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.754444 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" podStartSLOduration=1.7544336839999999 podStartE2EDuration="1.754433684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:02.754154531 +0000 UTC m=+1446.477581725" watchObservedRunningTime="2026-10-30 09:18:02.754433684 +0000 UTC m=+1446.477860826" Oct 30 09:18:02.769313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.769286 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbh6\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.769459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.769410 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.839632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.839596 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:02.858822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.858784 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:02.885338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.885307 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:18:02.885622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.885595 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:18:02.886494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:02.886475 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:18:03.156414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.156386 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:18:03.156899 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:03.156535 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1cc4ba_1def_4d05_bf38_0caa04e46736.slice/crio-9ead25c6fffeba7fd9b6fc07e413fcbee91d5a586513d3743306c2fd56614011 WatchSource:0}: Error finding container 9ead25c6fffeba7fd9b6fc07e413fcbee91d5a586513d3743306c2fd56614011: Status 404 returned error can't find the container with id 9ead25c6fffeba7fd9b6fc07e413fcbee91d5a586513d3743306c2fd56614011 Oct 30 09:18:03.159212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.159180 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:03.159346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.159317 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:03.159384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.159364 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:03.726424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.726329 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" event={"ID":"1f1cc4ba-1def-4d05-bf38-0caa04e46736","Type":"ContainerStarted","Data":"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c"} Oct 30 09:18:03.726424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.726390 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" event={"ID":"1f1cc4ba-1def-4d05-bf38-0caa04e46736","Type":"ContainerStarted","Data":"9ead25c6fffeba7fd9b6fc07e413fcbee91d5a586513d3743306c2fd56614011"} Oct 30 09:18:03.728044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.728013 4920 generic.go:356] "Generic (PLEG): container finished" podID="e980f5ac-1335-4aa9-85d6-2218c3ec12b3" containerID="4415b407a21780fcc264ec20883bae7acf0264bf8aea6a7d2cb836943f354cab" exitCode=0 Oct 30 09:18:03.728178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.728065 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" event={"ID":"e980f5ac-1335-4aa9-85d6-2218c3ec12b3","Type":"ContainerDied","Data":"4415b407a21780fcc264ec20883bae7acf0264bf8aea6a7d2cb836943f354cab"} Oct 30 09:18:03.729855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.729832 4920 generic.go:356] "Generic (PLEG): container finished" podID="02d670a2-0b5e-4576-a954-4c0db0b92236" containerID="17d952c5c4053362ad62e8fe33b023bcfe50e1b9cc2954f79740adca1e6da8a7" exitCode=0 Oct 30 09:18:03.729957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.729859 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" event={"ID":"02d670a2-0b5e-4576-a954-4c0db0b92236","Type":"ContainerDied","Data":"17d952c5c4053362ad62e8fe33b023bcfe50e1b9cc2954f79740adca1e6da8a7"} Oct 30 09:18:03.775454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.775394 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podStartSLOduration=1.775370197 podStartE2EDuration="1.775370197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:03.765443154 +0000 UTC m=+1447.488870297" watchObservedRunningTime="2026-10-30 09:18:03.775370197 +0000 UTC m=+1447.498797342" Oct 30 09:18:03.840673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:03.840618 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:04.604146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:04.604105 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:04.845033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:04.842907 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:04.845033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:04.842970 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:04.983206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:04.983180 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:04.985296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:04.985276 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:05.089043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089012 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\" (UniqueName: \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\") pod \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " Oct 30 09:18:05.089234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089050 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\" (UniqueName: \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\") pod \"02d670a2-0b5e-4576-a954-4c0db0b92236\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " Oct 30 09:18:05.089234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089080 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\") pod \"02d670a2-0b5e-4576-a954-4c0db0b92236\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " Oct 30 09:18:05.089234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089099 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\") pod \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " Oct 30 09:18:05.089234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089206 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\") pod \"02d670a2-0b5e-4576-a954-4c0db0b92236\" (UID: \"02d670a2-0b5e-4576-a954-4c0db0b92236\") " Oct 30 09:18:05.089475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089273 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\") pod \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\" (UID: \"e980f5ac-1335-4aa9-85d6-2218c3ec12b3\") " Oct 30 09:18:05.089535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089482 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca" pod "02d670a2-0b5e-4576-a954-4c0db0b92236" (UID: "02d670a2-0b5e-4576-a954-4c0db0b92236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:05.089838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089763 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca" pod "e980f5ac-1335-4aa9-85d6-2218c3ec12b3" (UID: "e980f5ac-1335-4aa9-85d6-2218c3ec12b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:05.089838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.089812 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.091864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.091524 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca" pod "e980f5ac-1335-4aa9-85d6-2218c3ec12b3" (UID: "e980f5ac-1335-4aa9-85d6-2218c3ec12b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:05.091864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.091657 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq" pod "e980f5ac-1335-4aa9-85d6-2218c3ec12b3" (UID: "e980f5ac-1335-4aa9-85d6-2218c3ec12b3"). InnerVolumeSpecName "kube-api-access-nfrbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:05.091864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.091828 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca" pod "02d670a2-0b5e-4576-a954-4c0db0b92236" (UID: "02d670a2-0b5e-4576-a954-4c0db0b92236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:05.092025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.091932 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz" pod "02d670a2-0b5e-4576-a954-4c0db0b92236" (UID: "02d670a2-0b5e-4576-a954-4c0db0b92236"). InnerVolumeSpecName "kube-api-access-2nlkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:05.190874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.190837 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.190874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.190866 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrbq\" (UniqueName: \"kubernetes.io/projected/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-kube-api-access-nfrbq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.190874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.190877 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nlkz\" (UniqueName: \"kubernetes.io/projected/02d670a2-0b5e-4576-a954-4c0db0b92236-kube-api-access-2nlkz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.190874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.190885 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e980f5ac-1335-4aa9-85d6-2218c3ec12b3-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.190874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.190894 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d670a2-0b5e-4576-a954-4c0db0b92236-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:05.744957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.744930 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:18:05.745438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.745120 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" containerID="cri-o://41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8" gracePeriod=30 Oct 30 09:18:05.751576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.751541 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" event={"ID":"e980f5ac-1335-4aa9-85d6-2218c3ec12b3","Type":"ContainerDied","Data":"1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae"} Oct 30 09:18:05.751748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.751601 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2f61b3b685aba8fa5fdf3f95ee90d5bcaa96b31d40660226f160416966a2ae" Oct 30 09:18:05.751883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.751860 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt" Oct 30 09:18:05.753382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.753359 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" Oct 30 09:18:05.753494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.753390 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm" event={"ID":"02d670a2-0b5e-4576-a954-4c0db0b92236","Type":"ContainerDied","Data":"871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68"} Oct 30 09:18:05.753494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.753437 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="871f7768c7d8046c695f21764a3381217c67e944ac6a53f736e1d519708eff68" Oct 30 09:18:05.840653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.840399 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:05.840653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:05.840460 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:06.153580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:06.153544 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:06.840499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:06.840461 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:06.840999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:06.840524 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:07.658517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:07.658475 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:07.840785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:07.840742 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:07.841210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:07.840812 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:07.992838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:07.992762 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:18:08.840376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:08.840336 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:08.840554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:08.840390 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:09.753365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:09.753300 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:09.843613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:09.843578 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:09.843939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:09.843896 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:10.850252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:10.848426 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 09:18:10.850252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:10.848515 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 09:18:10.888238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:10.887067 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 09:18:10.930415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:10.930363 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:11.037451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.037406 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:18:11.356303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.338489 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:18:11.534984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.532568 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:18:11.702251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702134 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702208 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702276 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702338 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702374 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702428 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702524 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702566 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702603 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\") pod \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\" (UID: \"1dda1e71-7a9c-4734-929b-f39c8c72a42d\") " Oct 30 09:18:11.702942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.702851 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:18:11.703252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703006 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:18:11.703252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703235 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:18:11.703396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703349 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.703396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703368 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.703501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703421 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.703501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703438 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:18:11.703879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.703855 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:11.705790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.705754 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:18:11.705858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.705834 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:11.705893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.705857 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "kube-api-access-g5gbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:11.709341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.709315 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo" pod "1dda1e71-7a9c-4734-929b-f39c8c72a42d" (UID: "1dda1e71-7a9c-4734-929b-f39c8c72a42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:18:11.804756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804726 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.804756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804754 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gbg\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-kube-api-access-g5gbg\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.804990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804764 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.804990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804775 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.804990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804782 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.804990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.804790 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dda1e71-7a9c-4734-929b-f39c8c72a42d-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:11.836473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.836434 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:11.874765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.874732 4920 generic.go:356] "Generic (PLEG): container finished" podID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerID="41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8" exitCode=0 Oct 30 09:18:11.875243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.874804 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" Oct 30 09:18:11.875243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.874820 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" event={"ID":"1dda1e71-7a9c-4734-929b-f39c8c72a42d","Type":"ContainerDied","Data":"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8"} Oct 30 09:18:11.875243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.874890 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5" event={"ID":"1dda1e71-7a9c-4734-929b-f39c8c72a42d","Type":"ContainerDied","Data":"694a47b85874d58b949386f35dbade0463faf77c8f8b8e4182702b2ae6417e9a"} Oct 30 09:18:11.875243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.874928 4920 scope.go:122] "RemoveContainer" containerID="41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8" Oct 30 09:18:11.892719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.892283 4920 scope.go:122] "RemoveContainer" containerID="41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8" Oct 30 09:18:11.893456 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:18:11.893382 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8\": container with ID starting with 41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8 not found: ID does not exist" containerID="41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8" Oct 30 09:18:11.893456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.893421 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8"} err="failed to get container status \"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8\": rpc error: code = NotFound desc = could not find container \"41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8\": container with ID starting with 41cdfdad4ce282234f147bb285a8f699704ebab8d54166a0e38d47454e2f10e8 not found: ID does not exist" Oct 30 09:18:11.935451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.935416 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:18:11.941090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:11.941053 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5krd5"] Oct 30 09:18:12.030705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.030606 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7"] Oct 30 09:18:12.031239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031204 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" Oct 30 09:18:12.031345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031243 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" Oct 30 09:18:12.031345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031261 4920 cpu_manager.go:414] "containerMap: removing container" podUID="02d670a2-0b5e-4576-a954-4c0db0b92236" containerName="revoke-keys" Oct 30 09:18:12.031345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031267 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d670a2-0b5e-4576-a954-4c0db0b92236" containerName="revoke-keys" Oct 30 09:18:12.031345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031281 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e980f5ac-1335-4aa9-85d6-2218c3ec12b3" containerName="revoke-keys" Oct 30 09:18:12.031345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031289 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e980f5ac-1335-4aa9-85d6-2218c3ec12b3" containerName="revoke-keys" Oct 30 09:18:12.031583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031487 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" containerName="istio-proxy" Oct 30 09:18:12.031583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031501 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d670a2-0b5e-4576-a954-4c0db0b92236" containerName="revoke-keys" Oct 30 09:18:12.031583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.031513 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e980f5ac-1335-4aa9-85d6-2218c3ec12b3" containerName="revoke-keys" Oct 30 09:18:12.036857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.036826 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.054487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.054450 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:12.058327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.058298 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:12.058470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.058424 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:12.066662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.066628 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7"] Oct 30 09:18:12.111645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.111593 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.111855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.111662 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.111855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.111709 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7sn\" (UniqueName: \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.154371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.154332 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:12.212585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.212539 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7sn\" (UniqueName: \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.212770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.212730 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.212827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.212776 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.213655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.213627 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.215342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.215315 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.233428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.233396 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7sn\" (UniqueName: \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\") pod \"maas-api-revoke-sub-b316fb13-pc7l7\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.355059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.355021 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:12.398815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.398781 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dda1e71-7a9c-4734-929b-f39c8c72a42d" path="/var/lib/kubelet/pods/1dda1e71-7a9c-4734-929b-f39c8c72a42d/volumes" Oct 30 09:18:12.402600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.400510 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs"] Oct 30 09:18:12.407255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.406736 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.430254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.430201 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs"] Oct 30 09:18:12.519154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.519106 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.519349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.519311 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.519545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.519513 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlbcd\" (UniqueName: \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.621495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.621406 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.621669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.621524 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.621669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.621629 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlbcd\" (UniqueName: \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.622788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.622756 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.624602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.624577 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.643273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.643244 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlbcd\" (UniqueName: \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\") pod \"maas-api-revoke-sub-b1f51fb8-6sxjs\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.665285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.665164 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7"] Oct 30 09:18:12.667992 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:12.667964 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80620d7_2367_44fd_9174_999c48e3c968.slice/crio-2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64 WatchSource:0}: Error finding container 2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64: Status 404 returned error can't find the container with id 2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64 Oct 30 09:18:12.735011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.734960 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:12.889306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.886575 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" event={"ID":"d80620d7-2367-44fd-9174-999c48e3c968","Type":"ContainerStarted","Data":"2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64"} Oct 30 09:18:12.889306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.887327 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:12.889306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:12.887366 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:13.067206 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:13.067171 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eeb6682_2b23_4593_ad17_ac8cf31f3057.slice/crio-20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb WatchSource:0}: Error finding container 20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb: Status 404 returned error can't find the container with id 20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb Oct 30 09:18:13.068190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.068166 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs"] Oct 30 09:18:13.209635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.209602 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:13.210096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.210066 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:13.216265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.216241 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:18:13.712792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.712759 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:13.896313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.896276 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" event={"ID":"3eeb6682-2b23-4593-ad17-ac8cf31f3057","Type":"ContainerStarted","Data":"0397bc6dc72a7c7c2f8a4a72a89c8a2af72e6d36dfefc93c37489c8935ffa37a"} Oct 30 09:18:13.896779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.896338 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" event={"ID":"3eeb6682-2b23-4593-ad17-ac8cf31f3057","Type":"ContainerStarted","Data":"20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb"} Oct 30 09:18:13.898547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.898514 4920 generic.go:356] "Generic (PLEG): container finished" podID="d80620d7-2367-44fd-9174-999c48e3c968" containerID="f1d7952602cd6a6936eb4065eabfc938cd2d9294a15eb7ab18eb3c54efeb0037" exitCode=0 Oct 30 09:18:13.898694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.898614 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" event={"ID":"d80620d7-2367-44fd-9174-999c48e3c968","Type":"ContainerDied","Data":"f1d7952602cd6a6936eb4065eabfc938cd2d9294a15eb7ab18eb3c54efeb0037"} Oct 30 09:18:13.943887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:13.943839 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" podStartSLOduration=1.9438243800000001 podStartE2EDuration="1.94382438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:13.940016126 +0000 UTC m=+1457.663443282" watchObservedRunningTime="2026-10-30 09:18:13.94382438 +0000 UTC m=+1457.667251522" Oct 30 09:18:14.907387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:14.907351 4920 generic.go:356] "Generic (PLEG): container finished" podID="3eeb6682-2b23-4593-ad17-ac8cf31f3057" containerID="0397bc6dc72a7c7c2f8a4a72a89c8a2af72e6d36dfefc93c37489c8935ffa37a" exitCode=0 Oct 30 09:18:14.907895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:14.907431 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" event={"ID":"3eeb6682-2b23-4593-ad17-ac8cf31f3057","Type":"ContainerDied","Data":"0397bc6dc72a7c7c2f8a4a72a89c8a2af72e6d36dfefc93c37489c8935ffa37a"} Oct 30 09:18:15.087114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.087087 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:15.154869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.154825 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\") pod \"d80620d7-2367-44fd-9174-999c48e3c968\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " Oct 30 09:18:15.155057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.155014 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\") pod \"d80620d7-2367-44fd-9174-999c48e3c968\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " Oct 30 09:18:15.155142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.155079 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\" (UniqueName: \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\") pod \"d80620d7-2367-44fd-9174-999c48e3c968\" (UID: \"d80620d7-2367-44fd-9174-999c48e3c968\") " Oct 30 09:18:15.155287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.155262 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca" pod "d80620d7-2367-44fd-9174-999c48e3c968" (UID: "d80620d7-2367-44fd-9174-999c48e3c968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:15.155506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.155485 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:15.157921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.157858 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn" pod "d80620d7-2367-44fd-9174-999c48e3c968" (UID: "d80620d7-2367-44fd-9174-999c48e3c968"). InnerVolumeSpecName "kube-api-access-bl7sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:15.157921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.157866 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca" pod "d80620d7-2367-44fd-9174-999c48e3c968" (UID: "d80620d7-2367-44fd-9174-999c48e3c968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:15.256688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.256650 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80620d7-2367-44fd-9174-999c48e3c968-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:15.256688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.256679 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl7sn\" (UniqueName: \"kubernetes.io/projected/d80620d7-2367-44fd-9174-999c48e3c968-kube-api-access-bl7sn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:15.575488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.575446 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:15.915955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.915870 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" Oct 30 09:18:15.915955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.915894 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7" event={"ID":"d80620d7-2367-44fd-9174-999c48e3c968","Type":"ContainerDied","Data":"2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64"} Oct 30 09:18:15.915955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:15.915946 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2beb7c7b3d0c272631e7fa773ff767ad8ffab42cb6497ea32e1aceaa1aad9d64" Oct 30 09:18:16.087649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.087623 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:16.168931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.168837 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\" (UniqueName: \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\") pod \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " Oct 30 09:18:16.168931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.168894 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\") pod \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " Oct 30 09:18:16.169154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.169043 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\") pod \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\" (UID: \"3eeb6682-2b23-4593-ad17-ac8cf31f3057\") " Oct 30 09:18:16.169444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.169421 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca" pod "3eeb6682-2b23-4593-ad17-ac8cf31f3057" (UID: "3eeb6682-2b23-4593-ad17-ac8cf31f3057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:16.169625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.169567 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:16.171053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.171032 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca" pod "3eeb6682-2b23-4593-ad17-ac8cf31f3057" (UID: "3eeb6682-2b23-4593-ad17-ac8cf31f3057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:16.171487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.171472 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd" pod "3eeb6682-2b23-4593-ad17-ac8cf31f3057" (UID: "3eeb6682-2b23-4593-ad17-ac8cf31f3057"). InnerVolumeSpecName "kube-api-access-jlbcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:16.270382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.270351 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlbcd\" (UniqueName: \"kubernetes.io/projected/3eeb6682-2b23-4593-ad17-ac8cf31f3057-kube-api-access-jlbcd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:16.270382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.270378 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eeb6682-2b23-4593-ad17-ac8cf31f3057-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:16.925436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.925399 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" event={"ID":"3eeb6682-2b23-4593-ad17-ac8cf31f3057","Type":"ContainerDied","Data":"20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb"} Oct 30 09:18:16.925942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.925434 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs" Oct 30 09:18:16.925942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:16.925450 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20584f2af6560da5578c118453dec96fd77d9a1408e43737228beabb31d2d9eb" Oct 30 09:18:17.451048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:17.451016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:18.690652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:18.690616 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:19.202436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:19.202399 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:20.687329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:20.687291 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:21.090809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:21.090768 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:22.854993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:22.854955 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:24.352927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.352887 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:18:24.353780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.353751 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3eeb6682-2b23-4593-ad17-ac8cf31f3057" containerName="revoke-keys" Oct 30 09:18:24.353934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.353913 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeb6682-2b23-4593-ad17-ac8cf31f3057" containerName="revoke-keys" Oct 30 09:18:24.354057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.353950 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d80620d7-2367-44fd-9174-999c48e3c968" containerName="revoke-keys" Oct 30 09:18:24.354057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.353958 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80620d7-2367-44fd-9174-999c48e3c968" containerName="revoke-keys" Oct 30 09:18:24.354178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.354131 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eeb6682-2b23-4593-ad17-ac8cf31f3057" containerName="revoke-keys" Oct 30 09:18:24.354178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.354144 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80620d7-2367-44fd-9174-999c48e3c968" containerName="revoke-keys" Oct 30 09:18:24.358380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.358358 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.381824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.381794 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:18:24.388415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.388386 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"openshift-service-ca.crt\"" Oct 30 09:18:24.388564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.388495 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"distinct-2-w3-176b6c-kserve-self-signed-certs\"" Oct 30 09:18:24.388668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.388650 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"kube-root-ca.crt\"" Oct 30 09:18:24.389730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.389710 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"default-dockercfg-z2c5q\"" Oct 30 09:18:24.430850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.430822 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:24.460047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.459973 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v"] Oct 30 09:18:24.464108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.464086 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.471582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.471547 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.471820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.471740 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.471820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.471795 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.471943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.471880 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.472100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.472074 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxnt\" (UniqueName: \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.472165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.472148 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.479229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.479199 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:24.479354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.479280 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:24.479354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.479288 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:24.479436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.479398 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:24.486038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.486012 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v"] Oct 30 09:18:24.556765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.556706 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:24.573105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573073 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573138 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.573305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573172 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573341 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573393 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573419 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573499 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573507 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxnt\" (UniqueName: \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573591 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.573709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573594 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.573709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573668 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndmc\" (UniqueName: \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.573773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.573699 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.575834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.575804 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.576037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.576019 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.596087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.596056 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxnt\" (UniqueName: \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\") pod \"distinct-2-w3-176b6c-kserve-cb49598bb-mb424\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.674285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.674185 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:24.674590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.674565 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndmc\" (UniqueName: \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.674649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.674637 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.674801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.674790 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.675451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.675427 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.676947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.676923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.709055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.709023 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndmc\" (UniqueName: \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\") pod \"maas-api-revoke-sub-97930d8f-p9l7v\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.746140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.746093 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:18:24.753250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.752850 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.780495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.780457 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:24.784513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.784481 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:18:24.876910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.876870 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.876925 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877058 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kdb\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877153 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877180 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877235 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877282 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.877416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.877353 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978418 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978518 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978690 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978738 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978794 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kdb\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978857 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.978881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978864 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978886 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978921 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978915 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.978944 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.979084 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.979308 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.979589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.979555 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.981712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.981687 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.983901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.981862 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:24.985386 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:24.985363 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cdc64f0_df08_435a_8583_81fed012ba0f.slice/crio-138dfa6f940500b478a3c22a9a3bc34aca44ad58fafe486affc8a02eea19435a WatchSource:0}: Error finding container 138dfa6f940500b478a3c22a9a3bc34aca44ad58fafe486affc8a02eea19435a: Status 404 returned error can't find the container with id 138dfa6f940500b478a3c22a9a3bc34aca44ad58fafe486affc8a02eea19435a Oct 30 09:18:24.987760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:24.987737 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:18:25.002728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.002699 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kdb\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:25.003702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.003673 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:25.071278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.071243 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:25.097446 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:25.097411 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2ceac0_04fd_4732_a294_aeeebcde7fff.slice/crio-0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35 WatchSource:0}: Error finding container 0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35: Status 404 returned error can't find the container with id 0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35 Oct 30 09:18:25.099904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.099643 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v"] Oct 30 09:18:25.414298 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:25.414271 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb406a82_75ea_427e_97b0_ac5c9805479c.slice/crio-2c29e40cad8276bb5eb2f310a60ff37cc715e519ae63bedec6fb4a59a57554c6 WatchSource:0}: Error finding container 2c29e40cad8276bb5eb2f310a60ff37cc715e519ae63bedec6fb4a59a57554c6: Status 404 returned error can't find the container with id 2c29e40cad8276bb5eb2f310a60ff37cc715e519ae63bedec6fb4a59a57554c6 Oct 30 09:18:25.415998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.415973 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:18:25.417568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.417536 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:25.417654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.417611 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:25.417710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:25.417657 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:26.036239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.035721 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" event={"ID":"7cdc64f0-df08-435a-8583-81fed012ba0f","Type":"ContainerStarted","Data":"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f"} Oct 30 09:18:26.036239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.035779 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" event={"ID":"7cdc64f0-df08-435a-8583-81fed012ba0f","Type":"ContainerStarted","Data":"138dfa6f940500b478a3c22a9a3bc34aca44ad58fafe486affc8a02eea19435a"} Oct 30 09:18:26.091050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.090255 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" event={"ID":"dd2ceac0-04fd-4732-a294-aeeebcde7fff","Type":"ContainerStarted","Data":"8d0d79217a6c860f034e07fc218fffa8556cee8975184a66285bfd5e23f2023b"} Oct 30 09:18:26.091050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.090323 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" event={"ID":"dd2ceac0-04fd-4732-a294-aeeebcde7fff","Type":"ContainerStarted","Data":"0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35"} Oct 30 09:18:26.093742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.093709 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" event={"ID":"fb406a82-75ea-427e-97b0-ac5c9805479c","Type":"ContainerStarted","Data":"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71"} Oct 30 09:18:26.093945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.093927 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" event={"ID":"fb406a82-75ea-427e-97b0-ac5c9805479c","Type":"ContainerStarted","Data":"2c29e40cad8276bb5eb2f310a60ff37cc715e519ae63bedec6fb4a59a57554c6"} Oct 30 09:18:26.113966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.113937 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:26.138547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.138489 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podStartSLOduration=2.138467396 podStartE2EDuration="2.138467396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:26.114366473 +0000 UTC m=+1469.837793635" watchObservedRunningTime="2026-10-30 09:18:26.138467396 +0000 UTC m=+1469.861894541" Oct 30 09:18:26.140309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.140280 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:26.140521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.140496 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:26.437302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.395887 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" podStartSLOduration=2.395866538 podStartE2EDuration="2.395866538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:26.178908997 +0000 UTC m=+1469.902336171" watchObservedRunningTime="2026-10-30 09:18:26.395866538 +0000 UTC m=+1470.119293681" Oct 30 09:18:26.550241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.537992 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podStartSLOduration=2.537972287 podStartE2EDuration="2.537972287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:26.453001424 +0000 UTC m=+1470.176428571" watchObservedRunningTime="2026-10-30 09:18:26.537972287 +0000 UTC m=+1470.261399431" Oct 30 09:18:26.884259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.801925 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:26.884679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.884640 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g"] Oct 30 09:18:26.988265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:26.970918 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.057447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.057408 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g"] Oct 30 09:18:27.073315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.072121 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:27.073315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.072174 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:27.159293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.152577 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r26m\" (UniqueName: \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.159293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.152670 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.159293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.152732 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.159293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.155776 4920 generic.go:356] "Generic (PLEG): container finished" podID="dd2ceac0-04fd-4732-a294-aeeebcde7fff" containerID="8d0d79217a6c860f034e07fc218fffa8556cee8975184a66285bfd5e23f2023b" exitCode=0 Oct 30 09:18:27.159293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.156925 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" event={"ID":"dd2ceac0-04fd-4732-a294-aeeebcde7fff","Type":"ContainerDied","Data":"8d0d79217a6c860f034e07fc218fffa8556cee8975184a66285bfd5e23f2023b"} Oct 30 09:18:27.219273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.216372 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:27.281254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.255986 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.281254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.256106 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.281254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.256293 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r26m\" (UniqueName: \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.281254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.273862 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.357240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.356527 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:18:27.386242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.383466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.413364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.413282 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r26m\" (UniqueName: \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\") pod \"maas-api-revoke-sub-22a05867-wnp7g\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.441365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.441313 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:27.460808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.460773 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:27.462677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.461435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:27.851397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.851335 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:27.851571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:27.851399 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:28.050085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.019313 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:28.077356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.072674 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:28.077356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.072739 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:28.181291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.180106 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9"] Oct 30 09:18:28.193088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.187331 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.193088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.193000 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9"] Oct 30 09:18:28.284561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.282623 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.284561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.282717 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.284561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.282794 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95b74\" (UniqueName: \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.352035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.352003 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g"] Oct 30 09:18:28.358635 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:28.357767 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67df86d7_bff6_46df_a181_d2a7f18af62c.slice/crio-24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef WatchSource:0}: Error finding container 24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef: Status 404 returned error can't find the container with id 24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef Oct 30 09:18:28.384335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.384301 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.384555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.384390 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.384555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.384439 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95b74\" (UniqueName: \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.385843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.385821 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.388808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.388759 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.422577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.415431 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95b74\" (UniqueName: \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\") pod \"maas-api-revoke-sub-d87edf0d-tmxn9\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.472370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.472346 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:28.485722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.485698 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\") pod \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " Oct 30 09:18:28.485842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.485799 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\") pod \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " Oct 30 09:18:28.485842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.485821 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\" (UniqueName: \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\") pod \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\" (UID: \"dd2ceac0-04fd-4732-a294-aeeebcde7fff\") " Oct 30 09:18:28.486118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.486092 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca" pod "dd2ceac0-04fd-4732-a294-aeeebcde7fff" (UID: "dd2ceac0-04fd-4732-a294-aeeebcde7fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:28.486251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.486234 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:28.487849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.487826 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca" pod "dd2ceac0-04fd-4732-a294-aeeebcde7fff" (UID: "dd2ceac0-04fd-4732-a294-aeeebcde7fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:28.487933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.487888 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc" pod "dd2ceac0-04fd-4732-a294-aeeebcde7fff" (UID: "dd2ceac0-04fd-4732-a294-aeeebcde7fff"). InnerVolumeSpecName "kube-api-access-dndmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:28.539206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.539166 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:28.587620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.587591 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2ceac0-04fd-4732-a294-aeeebcde7fff-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:28.587620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.587615 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndmc\" (UniqueName: \"kubernetes.io/projected/dd2ceac0-04fd-4732-a294-aeeebcde7fff-kube-api-access-dndmc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:28.918549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:28.917916 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9"] Oct 30 09:18:28.928033 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:28.928001 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68386fd_f97d_42a8_891f_7631f7bec75f.slice/crio-a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10 WatchSource:0}: Error finding container a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10: Status 404 returned error can't find the container with id a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10 Oct 30 09:18:29.072985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.072902 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:29.072985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.072958 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:29.232526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.229325 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" event={"ID":"dd2ceac0-04fd-4732-a294-aeeebcde7fff","Type":"ContainerDied","Data":"0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35"} Oct 30 09:18:29.232526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.229432 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac06c0ad025cff5d1f70d84cba93285f89024778f23c161f6b2f7091c142d35" Oct 30 09:18:29.232526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.229543 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v" Oct 30 09:18:29.234584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.233012 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" event={"ID":"67df86d7-bff6-46df-a181-d2a7f18af62c","Type":"ContainerStarted","Data":"5ffb1427ec4982bee5117568b0ca196e9fc9e81603584c4b0fac5c309bae59d1"} Oct 30 09:18:29.234584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.233069 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" event={"ID":"67df86d7-bff6-46df-a181-d2a7f18af62c","Type":"ContainerStarted","Data":"24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef"} Oct 30 09:18:29.245752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.245722 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" event={"ID":"b68386fd-f97d-42a8-891f-7631f7bec75f","Type":"ContainerStarted","Data":"a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10"} Oct 30 09:18:29.283636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.283574 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" podStartSLOduration=3.283555799 podStartE2EDuration="3.283555799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:29.277981085 +0000 UTC m=+1473.001408225" watchObservedRunningTime="2026-10-30 09:18:29.283555799 +0000 UTC m=+1473.006982962" Oct 30 09:18:29.489232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:29.484052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:30.071642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.071608 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:30.071810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.071661 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:30.255844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.255764 4920 generic.go:356] "Generic (PLEG): container finished" podID="b68386fd-f97d-42a8-891f-7631f7bec75f" containerID="51ff463990be73fd9d81e2eaa01d6576598fdb757c87459348146b231d7ac456" exitCode=0 Oct 30 09:18:30.256000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.255846 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" event={"ID":"b68386fd-f97d-42a8-891f-7631f7bec75f","Type":"ContainerDied","Data":"51ff463990be73fd9d81e2eaa01d6576598fdb757c87459348146b231d7ac456"} Oct 30 09:18:30.257586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.257561 4920 generic.go:356] "Generic (PLEG): container finished" podID="67df86d7-bff6-46df-a181-d2a7f18af62c" containerID="5ffb1427ec4982bee5117568b0ca196e9fc9e81603584c4b0fac5c309bae59d1" exitCode=0 Oct 30 09:18:30.257699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:30.257646 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" event={"ID":"67df86d7-bff6-46df-a181-d2a7f18af62c","Type":"ContainerDied","Data":"5ffb1427ec4982bee5117568b0ca196e9fc9e81603584c4b0fac5c309bae59d1"} Oct 30 09:18:31.071986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.071952 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:31.072395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.072008 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:31.147267 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:18:31.147212 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:18:31.256849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.256821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:31.497085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.497059 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:31.549786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.549746 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\" (UniqueName: \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\") pod \"b68386fd-f97d-42a8-891f-7631f7bec75f\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " Oct 30 09:18:31.552726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.552088 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\") pod \"b68386fd-f97d-42a8-891f-7631f7bec75f\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " Oct 30 09:18:31.553108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.552701 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca" pod "b68386fd-f97d-42a8-891f-7631f7bec75f" (UID: "b68386fd-f97d-42a8-891f-7631f7bec75f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:31.553294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.553276 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\") pod \"b68386fd-f97d-42a8-891f-7631f7bec75f\" (UID: \"b68386fd-f97d-42a8-891f-7631f7bec75f\") " Oct 30 09:18:31.554400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.554380 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.556126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.556064 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74" pod "b68386fd-f97d-42a8-891f-7631f7bec75f" (UID: "b68386fd-f97d-42a8-891f-7631f7bec75f"). InnerVolumeSpecName "kube-api-access-95b74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:31.556126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.556086 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca" pod "b68386fd-f97d-42a8-891f-7631f7bec75f" (UID: "b68386fd-f97d-42a8-891f-7631f7bec75f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:31.583197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.583171 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:31.655865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.655772 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\" (UniqueName: \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\") pod \"67df86d7-bff6-46df-a181-d2a7f18af62c\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " Oct 30 09:18:31.656030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.655866 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\") pod \"67df86d7-bff6-46df-a181-d2a7f18af62c\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " Oct 30 09:18:31.656030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.655934 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\") pod \"67df86d7-bff6-46df-a181-d2a7f18af62c\" (UID: \"67df86d7-bff6-46df-a181-d2a7f18af62c\") " Oct 30 09:18:31.656299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.656281 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95b74\" (UniqueName: \"kubernetes.io/projected/b68386fd-f97d-42a8-891f-7631f7bec75f-kube-api-access-95b74\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.656299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.656300 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68386fd-f97d-42a8-891f-7631f7bec75f-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.656554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.656538 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca" pod "67df86d7-bff6-46df-a181-d2a7f18af62c" (UID: "67df86d7-bff6-46df-a181-d2a7f18af62c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:31.658191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.658165 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca" pod "67df86d7-bff6-46df-a181-d2a7f18af62c" (UID: "67df86d7-bff6-46df-a181-d2a7f18af62c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:31.658412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.658389 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m" pod "67df86d7-bff6-46df-a181-d2a7f18af62c" (UID: "67df86d7-bff6-46df-a181-d2a7f18af62c"). InnerVolumeSpecName "kube-api-access-4r26m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:31.757287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.757252 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r26m\" (UniqueName: \"kubernetes.io/projected/67df86d7-bff6-46df-a181-d2a7f18af62c-kube-api-access-4r26m\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.757287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.757286 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.757492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.757298 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df86d7-bff6-46df-a181-d2a7f18af62c-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:31.908732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.908638 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf"] Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910188 4920 cpu_manager.go:414] "containerMap: removing container" podUID="dd2ceac0-04fd-4732-a294-aeeebcde7fff" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910238 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2ceac0-04fd-4732-a294-aeeebcde7fff" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910263 4920 cpu_manager.go:414] "containerMap: removing container" podUID="67df86d7-bff6-46df-a181-d2a7f18af62c" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910272 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67df86d7-bff6-46df-a181-d2a7f18af62c" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910315 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b68386fd-f97d-42a8-891f-7631f7bec75f" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910324 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68386fd-f97d-42a8-891f-7631f7bec75f" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910457 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2ceac0-04fd-4732-a294-aeeebcde7fff" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910472 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="67df86d7-bff6-46df-a181-d2a7f18af62c" containerName="revoke-keys" Oct 30 09:18:31.912341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.910488 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68386fd-f97d-42a8-891f-7631f7bec75f" containerName="revoke-keys" Oct 30 09:18:31.917647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.917622 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:31.959520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.959480 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9255v\" (UniqueName: \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:31.959683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.959573 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:31.959788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.959718 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:31.971033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:31.969550 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf"] Oct 30 09:18:32.060788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.060744 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9255v\" (UniqueName: \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.060959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.060833 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.060959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.060890 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.061896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.061869 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.063316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.063295 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.072459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.072435 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:32.072776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.072491 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:32.080685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.080659 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9255v\" (UniqueName: \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\") pod \"maas-api-revoke-sub-d234355c-6d5lf\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.242171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.242085 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:32.276410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.275845 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" Oct 30 09:18:32.276410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.275849 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9" event={"ID":"b68386fd-f97d-42a8-891f-7631f7bec75f","Type":"ContainerDied","Data":"a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10"} Oct 30 09:18:32.276410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.275951 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a437752b60097d44feb1bd3ccaa2c525f396de8528dea827e8bbcf1976f32a10" Oct 30 09:18:32.277468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.277441 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" Oct 30 09:18:32.277596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.277489 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g" event={"ID":"67df86d7-bff6-46df-a181-d2a7f18af62c","Type":"ContainerDied","Data":"24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef"} Oct 30 09:18:32.277596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.277535 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ed37d1ce48135fafec544bfccf2118c78026f5f1763dc36229c03cd98aefef" Oct 30 09:18:32.604882 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:32.604845 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff673e7d_735f_46ba_afca_4e10cdb88057.slice/crio-417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61 WatchSource:0}: Error finding container 417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61: Status 404 returned error can't find the container with id 417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61 Oct 30 09:18:32.608901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.608869 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf"] Oct 30 09:18:32.956953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:32.956885 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:33.072400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.072361 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 09:18:33.072574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.072417 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 09:18:33.088076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.088039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:33.288345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.288257 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" event={"ID":"ff673e7d-735f-46ba-afca-4e10cdb88057","Type":"ContainerStarted","Data":"06a8677e0c113212866f6cbaddc21aab0fdbae2e3eed59ce85958523a5aef948"} Oct 30 09:18:33.288345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.288335 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" event={"ID":"ff673e7d-735f-46ba-afca-4e10cdb88057","Type":"ContainerStarted","Data":"417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61"} Oct 30 09:18:33.362560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:33.362506 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" podStartSLOduration=2.362490835 podStartE2EDuration="2.362490835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:33.360283758 +0000 UTC m=+1477.083710902" watchObservedRunningTime="2026-10-30 09:18:33.362490835 +0000 UTC m=+1477.085917977" Oct 30 09:18:34.076349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.076314 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:34.076749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.076729 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:34.077569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.077552 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:18:34.303387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.303354 4920 generic.go:356] "Generic (PLEG): container finished" podID="ff673e7d-735f-46ba-afca-4e10cdb88057" containerID="06a8677e0c113212866f6cbaddc21aab0fdbae2e3eed59ce85958523a5aef948" exitCode=0 Oct 30 09:18:34.303711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.303583 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" event={"ID":"ff673e7d-735f-46ba-afca-4e10cdb88057","Type":"ContainerDied","Data":"06a8677e0c113212866f6cbaddc21aab0fdbae2e3eed59ce85958523a5aef948"} Oct 30 09:18:34.674569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.674527 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:34.687170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:34.687141 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:35.016132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.016050 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:35.333422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.333391 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:35.481755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.481729 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:35.582312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.582237 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:18:35.582829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.582811 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ff673e7d-735f-46ba-afca-4e10cdb88057" containerName="revoke-keys" Oct 30 09:18:35.582829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.582829 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff673e7d-735f-46ba-afca-4e10cdb88057" containerName="revoke-keys" Oct 30 09:18:35.583010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.582995 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff673e7d-735f-46ba-afca-4e10cdb88057" containerName="revoke-keys" Oct 30 09:18:35.586727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.586706 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.614069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.614027 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\") pod \"ff673e7d-735f-46ba-afca-4e10cdb88057\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " Oct 30 09:18:35.614787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.614763 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\" (UniqueName: \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\") pod \"ff673e7d-735f-46ba-afca-4e10cdb88057\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " Oct 30 09:18:35.614962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.614946 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\") pod \"ff673e7d-735f-46ba-afca-4e10cdb88057\" (UID: \"ff673e7d-735f-46ba-afca-4e10cdb88057\") " Oct 30 09:18:35.615925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.615904 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca" pod "ff673e7d-735f-46ba-afca-4e10cdb88057" (UID: "ff673e7d-735f-46ba-afca-4e10cdb88057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:35.618512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.617121 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:18:35.635691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.635666 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca" pod "ff673e7d-735f-46ba-afca-4e10cdb88057" (UID: "ff673e7d-735f-46ba-afca-4e10cdb88057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:35.637962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.637933 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v" pod "ff673e7d-735f-46ba-afca-4e10cdb88057" (UID: "ff673e7d-735f-46ba-afca-4e10cdb88057"). InnerVolumeSpecName "kube-api-access-9255v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:35.717009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.716967 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717017 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717040 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717142 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj52\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717247 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717352 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717398 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717534 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717613 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.717777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717762 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9255v\" (UniqueName: \"kubernetes.io/projected/ff673e7d-735f-46ba-afca-4e10cdb88057-kube-api-access-9255v\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:35.717811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717783 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:35.717811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.717798 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff673e7d-735f-46ba-afca-4e10cdb88057-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:35.818977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.818938 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.818990 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819066 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819139 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819237 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819272 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819296 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819323 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj52\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819380 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819539 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819702 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819783 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.819907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.819880 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.820369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.820346 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.821673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.821651 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.821887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.821871 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.845000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.844927 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.845245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.845200 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj52\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:35.900105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:35.900071 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:36.224455 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:36.224416 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496e7447_55a8_416b_9a54_22ece43909c3.slice/crio-6603a281d99bf905fc864cb530e5ebbbbb73a19c89a5b96a3de162f95385e603 WatchSource:0}: Error finding container 6603a281d99bf905fc864cb530e5ebbbbb73a19c89a5b96a3de162f95385e603: Status 404 returned error can't find the container with id 6603a281d99bf905fc864cb530e5ebbbbb73a19c89a5b96a3de162f95385e603 Oct 30 09:18:36.226949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.226918 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.227058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.226981 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.227058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.227010 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.229473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.229384 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:18:36.329020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.328977 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" event={"ID":"496e7447-55a8-416b-9a54-22ece43909c3","Type":"ContainerStarted","Data":"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84"} Oct 30 09:18:36.329245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.329049 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" event={"ID":"496e7447-55a8-416b-9a54-22ece43909c3","Type":"ContainerStarted","Data":"6603a281d99bf905fc864cb530e5ebbbbb73a19c89a5b96a3de162f95385e603"} Oct 30 09:18:36.338531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.338500 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" Oct 30 09:18:36.339597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.339574 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf" event={"ID":"ff673e7d-735f-46ba-afca-4e10cdb88057","Type":"ContainerDied","Data":"417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61"} Oct 30 09:18:36.339691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.339627 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="417d984217eceb32a007a9eeacf7f60fa0d1c8292a80a2cb42cee54d4b6e5a61" Oct 30 09:18:36.394261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.394194 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podStartSLOduration=1.394175074 podStartE2EDuration="1.394175074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:36.392372527 +0000 UTC m=+1480.115799673" watchObservedRunningTime="2026-10-30 09:18:36.394175074 +0000 UTC m=+1480.117602219" Oct 30 09:18:36.599003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.598971 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:18:36.835844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.835807 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k"] Oct 30 09:18:36.840995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.840969 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:36.856648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.856236 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:36.856648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.856331 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:36.856648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.856387 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:36.856648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.856457 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k"] Oct 30 09:18:36.861681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.861660 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:36.921240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.900321 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:36.933617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.933438 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:36.933617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.933481 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:36.933617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:36.933523 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf7x\" (UniqueName: \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.020853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.020008 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:37.034659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.034626 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.034853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.034687 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.034853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.034750 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf7x\" (UniqueName: \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.035676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.035650 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.037732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.037711 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.059256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.059189 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf7x\" (UniqueName: \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\") pod \"maas-api-revoke-sub-08497a89-dhd5k\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.163921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.163833 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:37.536004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.533317 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k"] Oct 30 09:18:37.536004 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:37.535566 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4748a6d_905a_4fa3_96e5_da9b5c834ef0.slice/crio-adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8 WatchSource:0}: Error finding container adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8: Status 404 returned error can't find the container with id adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8 Oct 30 09:18:37.940347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.939510 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:37.940347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:37.939578 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:38.571240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.570970 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" event={"ID":"e4748a6d-905a-4fa3-96e5-da9b5c834ef0","Type":"ContainerStarted","Data":"2152008e62bbc23e132c1be1fc0e195b4922a5697c28e79ec4f9395169516d6b"} Oct 30 09:18:38.571240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.571034 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" event={"ID":"e4748a6d-905a-4fa3-96e5-da9b5c834ef0","Type":"ContainerStarted","Data":"adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8"} Oct 30 09:18:38.656254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.646165 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" podStartSLOduration=2.646141519 podStartE2EDuration="2.646141519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:38.614652792 +0000 UTC m=+1482.338079964" watchObservedRunningTime="2026-10-30 09:18:38.646141519 +0000 UTC m=+1482.369568663" Oct 30 09:18:38.909525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.908173 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.909525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.908251 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:38.926388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:38.926343 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.040396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.040345 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.170303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.169342 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.325243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.323637 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.442436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.442326 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.596972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.596935 4920 generic.go:356] "Generic (PLEG): container finished" podID="e4748a6d-905a-4fa3-96e5-da9b5c834ef0" containerID="2152008e62bbc23e132c1be1fc0e195b4922a5697c28e79ec4f9395169516d6b" exitCode=0 Oct 30 09:18:39.597804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.597765 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" event={"ID":"e4748a6d-905a-4fa3-96e5-da9b5c834ef0","Type":"ContainerDied","Data":"2152008e62bbc23e132c1be1fc0e195b4922a5697c28e79ec4f9395169516d6b"} Oct 30 09:18:40.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.953872 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:18:40.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.955505 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:18:40.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.964452 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:39.964505 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:40.043907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.043859 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.141269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.049502 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:18:40.168339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.168302 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:18:40.193273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.172134 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.365327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.341006 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:18:40.365327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.342511 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.488824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.488775 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:18:40.526265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.508140 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:18:40.526265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.510162 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744383 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744438 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744470 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744507 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744599 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964fn\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744647 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744680 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744770 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.754299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.744823 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.853720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.853683 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.854825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.854800 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.862337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.862308 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.889438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.889399 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.890073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.890050 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.890684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.890654 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964fn\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.891168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.891146 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.891663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.891645 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.891879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.891862 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.892059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.858554 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.892165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.889968 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.892277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.854679 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.914285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.893098 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:40.915192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:40.890508 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:41.009568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.009469 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.123573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.122327 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.186240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.183036 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:41.186240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.184172 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:41.211242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.186718 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.216345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.216299 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:41.355681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.187398 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 09:18:41.356055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.356015 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:18:41.409349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.366520 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.409349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.366890 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.470239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.456652 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:41.470239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.456716 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.470239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.464448 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.601241 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:18:41.523556 4920 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.015s" Oct 30 09:18:41.601241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.595463 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:18:41.601241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.595559 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:18:41.601241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.595597 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:18:41.601241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.595697 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:41.814241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.805301 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:41.948401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.892447 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:42.028243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:41.913893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-964fn\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:42.029280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.028574 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:18:42.029894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.029871 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:42.067372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.038147 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:18:42.068632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.068501 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:42.068781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.068673 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:42.070169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.070141 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:18:42.177712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.177687 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:18:42.177954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.177943 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:42.186519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.186496 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:18:42.229540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.229511 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:18:42.230044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.230028 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:18:42.379426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.379394 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:42.494199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.494166 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\") pod \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " Oct 30 09:18:42.494419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.494276 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\" (UniqueName: \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\") pod \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " Oct 30 09:18:42.494496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.494418 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\") pod \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\" (UID: \"e4748a6d-905a-4fa3-96e5-da9b5c834ef0\") " Oct 30 09:18:42.494561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.494540 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca" pod "e4748a6d-905a-4fa3-96e5-da9b5c834ef0" (UID: "e4748a6d-905a-4fa3-96e5-da9b5c834ef0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:42.494895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.494875 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:42.496859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.496829 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x" pod "e4748a6d-905a-4fa3-96e5-da9b5c834ef0" (UID: "e4748a6d-905a-4fa3-96e5-da9b5c834ef0"). InnerVolumeSpecName "kube-api-access-dxf7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:42.496984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.496874 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca" pod "e4748a6d-905a-4fa3-96e5-da9b5c834ef0" (UID: "e4748a6d-905a-4fa3-96e5-da9b5c834ef0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:42.596626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.596545 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:42.596837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.596820 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf7x\" (UniqueName: \"kubernetes.io/projected/e4748a6d-905a-4fa3-96e5-da9b5c834ef0-kube-api-access-dxf7x\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:42.635776 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:42.635742 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b7e1e5_4d24_4b50_b35e_359d1cceae42.slice/crio-9f0ab16be77160bdd73534b1c14afb47d27b93afc1be5c451ee8fc7770bc0af3 WatchSource:0}: Error finding container 9f0ab16be77160bdd73534b1c14afb47d27b93afc1be5c451ee8fc7770bc0af3: Status 404 returned error can't find the container with id 9f0ab16be77160bdd73534b1c14afb47d27b93afc1be5c451ee8fc7770bc0af3 Oct 30 09:18:42.639069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.639039 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:18:42.639514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.639478 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.639597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.639579 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.639647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.639628 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.903296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.902601 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:42.903296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:42.902662 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:43.252239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.249126 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" event={"ID":"e4748a6d-905a-4fa3-96e5-da9b5c834ef0","Type":"ContainerDied","Data":"adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8"} Oct 30 09:18:43.252239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.249191 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adebaec76949db6b44cb42466c968e3e5090ec91bc83a9392640f48d432314c8" Oct 30 09:18:43.252239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.249287 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k" Oct 30 09:18:43.267104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.267074 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" event={"ID":"a8b7e1e5-4d24-4b50-b35e-359d1cceae42","Type":"ContainerStarted","Data":"cba4633658200bb076c85f10d288bf17326ef498002adbaabd9741f51cc3b5e0"} Oct 30 09:18:43.268798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.268774 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" event={"ID":"a8b7e1e5-4d24-4b50-b35e-359d1cceae42","Type":"ContainerStarted","Data":"9f0ab16be77160bdd73534b1c14afb47d27b93afc1be5c451ee8fc7770bc0af3"} Oct 30 09:18:43.338621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.338555 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podStartSLOduration=4.338532857 podStartE2EDuration="4.338532857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:43.321517773 +0000 UTC m=+1487.044944916" watchObservedRunningTime="2026-10-30 09:18:43.338532857 +0000 UTC m=+1487.061960000" Oct 30 09:18:43.790492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.790459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:43.905242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.903658 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:43.905242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:43.903723 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:44.145303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.144552 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:44.152239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.147334 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.152239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.147408 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:44.894584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.894537 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl"] Oct 30 09:18:44.895129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.895114 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e4748a6d-905a-4fa3-96e5-da9b5c834ef0" containerName="revoke-keys" Oct 30 09:18:44.895129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.895129 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4748a6d-905a-4fa3-96e5-da9b5c834ef0" containerName="revoke-keys" Oct 30 09:18:44.895296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.895208 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4748a6d-905a-4fa3-96e5-da9b5c834ef0" containerName="revoke-keys" Oct 30 09:18:44.901241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.901095 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.901241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.901148 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 09:18:44.912240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.907513 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:44.933262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.932001 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:44.933262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.932079 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:44.937366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.937329 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl"] Oct 30 09:18:44.943380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.943329 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:44.943532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:44.943489 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:45.070781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.070746 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhk8\" (UniqueName: \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.070996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.070812 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.071064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.071046 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.144498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.144453 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:45.144723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.144511 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:45.171809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.171773 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.171969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.171897 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.171969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.171942 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhk8\" (UniqueName: \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.173034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.173015 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.175665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.175643 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.196292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.196264 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhk8\" (UniqueName: \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\") pod \"maas-api-revoke-sub-ef722d31-nbjsl\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.249523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.248452 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:45.598813 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:45.598782 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f57307_f893_4963_bffd_f23aa71b71c3.slice/crio-de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303 WatchSource:0}: Error finding container de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303: Status 404 returned error can't find the container with id de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303 Oct 30 09:18:45.626818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.617599 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl"] Oct 30 09:18:45.955246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:45.955108 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:46.143501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.143458 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:46.143691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.143530 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:46.162527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.162490 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8"] Oct 30 09:18:46.166956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.166931 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.196847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.196334 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8"] Oct 30 09:18:46.302240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.297361 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.313410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.302505 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsgc\" (UniqueName: \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.313689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.313585 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.361795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.361651 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" event={"ID":"f5f57307-f893-4963-bffd-f23aa71b71c3","Type":"ContainerStarted","Data":"4a62509078d2d71365af5994d646cf64cf6fc1980f99ba6488ffa3c9baef902d"} Oct 30 09:18:46.361795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.361756 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" event={"ID":"f5f57307-f893-4963-bffd-f23aa71b71c3","Type":"ContainerStarted","Data":"de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303"} Oct 30 09:18:46.422239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.417935 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.422239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.418005 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsgc\" (UniqueName: \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.422239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.418131 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.422239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.419899 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.444524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.444490 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.445634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.445406 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsgc\" (UniqueName: \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\") pod \"maas-api-revoke-sub-45af605c-mpvt8\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.445634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.445412 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" podStartSLOduration=2.445394381 podStartE2EDuration="2.445394381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:46.411488034 +0000 UTC m=+1490.134915195" watchObservedRunningTime="2026-10-30 09:18:46.445394381 +0000 UTC m=+1490.168821528" Oct 30 09:18:46.560071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.559963 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:46.936249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.936102 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:46.936249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.936158 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:46.941146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:46.941118 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8"] Oct 30 09:18:46.941685 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:46.941663 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1390ba2_5973_4111_bd72_c0da68514966.slice/crio-cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609 WatchSource:0}: Error finding container cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609: Status 404 returned error can't find the container with id cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609 Oct 30 09:18:47.147411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.147378 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:47.147875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.147435 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:47.377600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.377565 4920 generic.go:356] "Generic (PLEG): container finished" podID="f5f57307-f893-4963-bffd-f23aa71b71c3" containerID="4a62509078d2d71365af5994d646cf64cf6fc1980f99ba6488ffa3c9baef902d" exitCode=0 Oct 30 09:18:47.377948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.377651 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" event={"ID":"f5f57307-f893-4963-bffd-f23aa71b71c3","Type":"ContainerDied","Data":"4a62509078d2d71365af5994d646cf64cf6fc1980f99ba6488ffa3c9baef902d"} Oct 30 09:18:47.380236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.380177 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" event={"ID":"b1390ba2-5973-4111-bd72-c0da68514966","Type":"ContainerStarted","Data":"ef2a7d3f4b3dec81eb8b7aae3f33cddc54d509c2fb3d5b6048061812e80a73e1"} Oct 30 09:18:47.380355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.380273 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" event={"ID":"b1390ba2-5973-4111-bd72-c0da68514966","Type":"ContainerStarted","Data":"cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609"} Oct 30 09:18:47.480079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.479976 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" podStartSLOduration=1.479960929 podStartE2EDuration="1.479960929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:47.475102716 +0000 UTC m=+1491.198529856" watchObservedRunningTime="2026-10-30 09:18:47.479960929 +0000 UTC m=+1491.203388072" Oct 30 09:18:47.678543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.678506 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:18:47.682866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.682843 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.749881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.743392 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:18:47.753068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.750083 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:18:47.775660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.763545 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.783785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.783198 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:18:47.855285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855247 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.855285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855286 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855323 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855339 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855372 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2j7\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855398 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855435 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855459 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855479 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855503 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855519 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855536 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.855566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855558 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.856124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855598 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.856124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855623 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhdf\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.856124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855660 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.856124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855690 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.856124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.855713 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.897120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.897086 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:47.905318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.905273 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:47.922555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.919517 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:18:47.956479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956447 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.956678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956486 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.956678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956511 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.956936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956910 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956931 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956970 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.956977 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957063 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957094 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhdf\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957138 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957161 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957279 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957427 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957780 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957809 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.957886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957867 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.957886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957872 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.958114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.957890 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.958252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.958200 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2j7\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.959018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.958991 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.959116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.959083 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.959171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.959139 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.959559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.959531 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.959671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.959584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.960177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.960148 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.960629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.960611 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.960744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.960727 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.960858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.960837 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.961059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.961043 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.974532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.974380 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.975695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.975670 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.977349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.977329 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhdf\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.983467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.983374 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:47.983467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.983392 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:47.985736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:47.985705 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2j7\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:48.036314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.033550 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:48.036314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.035727 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:48.091676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.091350 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:48.145471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.145002 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:48.145471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.145060 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:48.392050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.391961 4920 generic.go:356] "Generic (PLEG): container finished" podID="b1390ba2-5973-4111-bd72-c0da68514966" containerID="ef2a7d3f4b3dec81eb8b7aae3f33cddc54d509c2fb3d5b6048061812e80a73e1" exitCode=0 Oct 30 09:18:48.426242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.406765 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" event={"ID":"b1390ba2-5973-4111-bd72-c0da68514966","Type":"ContainerDied","Data":"ef2a7d3f4b3dec81eb8b7aae3f33cddc54d509c2fb3d5b6048061812e80a73e1"} Oct 30 09:18:48.443694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.441858 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:18:48.447411 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:48.447376 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912d892e_fde1_47a5_bab8_6afe0468e0b4.slice/crio-1d7d90b7205343d02780fe68c9a113dc26562d1eaa3ca7fa33239f58859954f0 WatchSource:0}: Error finding container 1d7d90b7205343d02780fe68c9a113dc26562d1eaa3ca7fa33239f58859954f0: Status 404 returned error can't find the container with id 1d7d90b7205343d02780fe68c9a113dc26562d1eaa3ca7fa33239f58859954f0 Oct 30 09:18:48.453269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.450975 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.453269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.451069 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.453269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.451116 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.541777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.541734 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:18:48.563411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.562119 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.563411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.562254 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.563411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.562346 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.643289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.642329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:48.732874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.732844 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:48.785983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.784738 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\") pod \"f5f57307-f893-4963-bffd-f23aa71b71c3\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " Oct 30 09:18:48.785983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.784832 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\") pod \"f5f57307-f893-4963-bffd-f23aa71b71c3\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " Oct 30 09:18:48.785983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.785009 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\" (UniqueName: \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\") pod \"f5f57307-f893-4963-bffd-f23aa71b71c3\" (UID: \"f5f57307-f893-4963-bffd-f23aa71b71c3\") " Oct 30 09:18:48.788561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.788286 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8" pod "f5f57307-f893-4963-bffd-f23aa71b71c3" (UID: "f5f57307-f893-4963-bffd-f23aa71b71c3"). InnerVolumeSpecName "kube-api-access-frhk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:48.789714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.789695 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca" pod "f5f57307-f893-4963-bffd-f23aa71b71c3" (UID: "f5f57307-f893-4963-bffd-f23aa71b71c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:48.790644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.790607 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca" pod "f5f57307-f893-4963-bffd-f23aa71b71c3" (UID: "f5f57307-f893-4963-bffd-f23aa71b71c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:48.886615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.886581 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:48.886615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.886613 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5f57307-f893-4963-bffd-f23aa71b71c3-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:48.886615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:48.886622 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhk8\" (UniqueName: \"kubernetes.io/projected/f5f57307-f893-4963-bffd-f23aa71b71c3-kube-api-access-frhk8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:49.144386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.144340 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 09:18:49.144607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.144408 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 09:18:49.261401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.261367 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:18:49.261971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.261957 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f5f57307-f893-4963-bffd-f23aa71b71c3" containerName="revoke-keys" Oct 30 09:18:49.262010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.261972 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f57307-f893-4963-bffd-f23aa71b71c3" containerName="revoke-keys" Oct 30 09:18:49.262083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.262074 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f57307-f893-4963-bffd-f23aa71b71c3" containerName="revoke-keys" Oct 30 09:18:49.265919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.265898 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.292761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.292725 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.294761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.294729 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.294963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.294938 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.294994 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.295025 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2t9\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.295071 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.295113 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.295230 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.295392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.295336 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.305256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.305027 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:18:49.406296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406160 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406255 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406296 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406349 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406393 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2t9\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406442 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406499 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406623 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.406827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.406738 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.421237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.419185 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.421237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.419504 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.421237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.420711 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.421237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.420985 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.421567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.421281 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.451239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.450054 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.469097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.469051 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2t9\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.470261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.469559 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.474447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.473049 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" event={"ID":"f5f57307-f893-4963-bffd-f23aa71b71c3","Type":"ContainerDied","Data":"de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303"} Oct 30 09:18:49.474447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.473115 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de17af823ddf45aa2b8b72874d694cf12f8025d0218f18293de4b9346f07f303" Oct 30 09:18:49.474447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.473183 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl" Oct 30 09:18:49.492119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.478547 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.492119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.491139 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" event={"ID":"8a4a02e3-58c3-4309-ad17-b4bbc670a35e","Type":"ContainerStarted","Data":"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2"} Oct 30 09:18:49.492119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.491230 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" event={"ID":"8a4a02e3-58c3-4309-ad17-b4bbc670a35e","Type":"ContainerStarted","Data":"473318adfae39566d98877a83166b29435f45d64b59e9b121fc96a40ee26a7fa"} Oct 30 09:18:49.513255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.498301 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" event={"ID":"912d892e-fde1-47a5-bab8-6afe0468e0b4","Type":"ContainerStarted","Data":"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414"} Oct 30 09:18:49.513255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.498358 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" event={"ID":"912d892e-fde1-47a5-bab8-6afe0468e0b4","Type":"ContainerStarted","Data":"1d7d90b7205343d02780fe68c9a113dc26562d1eaa3ca7fa33239f58859954f0"} Oct 30 09:18:49.578276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.576794 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podStartSLOduration=2.576771289 podStartE2EDuration="2.576771289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:49.552742858 +0000 UTC m=+1493.276170003" watchObservedRunningTime="2026-10-30 09:18:49.576771289 +0000 UTC m=+1493.300198433" Oct 30 09:18:49.589258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.589028 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:49.685847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:49.685765 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podStartSLOduration=2.6857445970000002 podStartE2EDuration="2.685744597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:49.646613237 +0000 UTC m=+1493.370040405" watchObservedRunningTime="2026-10-30 09:18:49.685744597 +0000 UTC m=+1493.409171743" Oct 30 09:18:50.070728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.070602 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:18:50.089427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.087204 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:50.089427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.087300 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:50.104350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.094458 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:18:50.149238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.130708 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:50.149238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.130767 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:50.469306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.403011 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:50.469306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.405183 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:50.617244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.575736 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\") pod \"b1390ba2-5973-4111-bd72-c0da68514966\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " Oct 30 09:18:50.617244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.575830 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\") pod \"b1390ba2-5973-4111-bd72-c0da68514966\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " Oct 30 09:18:50.617244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.575979 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\" (UniqueName: \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\") pod \"b1390ba2-5973-4111-bd72-c0da68514966\" (UID: \"b1390ba2-5973-4111-bd72-c0da68514966\") " Oct 30 09:18:50.617244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.589628 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca" pod "b1390ba2-5973-4111-bd72-c0da68514966" (UID: "b1390ba2-5973-4111-bd72-c0da68514966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:18:50.693042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.668371 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca" pod "b1390ba2-5973-4111-bd72-c0da68514966" (UID: "b1390ba2-5973-4111-bd72-c0da68514966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:18:50.709284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.704268 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:18:50.709284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.706852 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:50.709284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.706943 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:50.709284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.706990 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:50.783937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.776292 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:50.783937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.776354 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1390ba2-5973-4111-bd72-c0da68514966-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:50.790369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.788182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc" pod "b1390ba2-5973-4111-bd72-c0da68514966" (UID: "b1390ba2-5973-4111-bd72-c0da68514966"). InnerVolumeSpecName "kube-api-access-hxsgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:18:50.790369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.788340 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" Oct 30 09:18:50.790369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.788650 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8" event={"ID":"b1390ba2-5973-4111-bd72-c0da68514966","Type":"ContainerDied","Data":"cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609"} Oct 30 09:18:50.790369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.788683 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff3f7712a670b20daf2c18a78b419afe2f4b59cd07c467bc4c211e380afb609" Oct 30 09:18:50.884931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.881926 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsgc\" (UniqueName: \"kubernetes.io/projected/b1390ba2-5973-4111-bd72-c0da68514966-kube-api-access-hxsgc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:18:50.931968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.931925 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:50.939628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.933640 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:50.958245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:50.953362 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:18:51.049243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.047701 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:51.049243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.047766 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:51.092265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.092231 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:51.092500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.092476 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:51.766580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.766540 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w"] Oct 30 09:18:51.767505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.767482 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b1390ba2-5973-4111-bd72-c0da68514966" containerName="revoke-keys" Oct 30 09:18:51.767602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.767507 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1390ba2-5973-4111-bd72-c0da68514966" containerName="revoke-keys" Oct 30 09:18:51.767692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.767680 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1390ba2-5973-4111-bd72-c0da68514966" containerName="revoke-keys" Oct 30 09:18:51.771916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.771897 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.805296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.797657 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.805296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.797729 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.805296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.797880 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pds\" (UniqueName: \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.814616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.814582 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:18:51.828244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.815449 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:18:51.828244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.817105 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:18:51.828244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.817209 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:18:51.828997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.828963 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" event={"ID":"42896e60-9b13-48e6-92ab-cb4778354f68","Type":"ContainerStarted","Data":"1a62bb890e0882659995fd45e249af3280045a88c8122802e12fe0ce6b4b29b1"} Oct 30 09:18:51.856120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.856081 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" event={"ID":"42896e60-9b13-48e6-92ab-cb4778354f68","Type":"ContainerStarted","Data":"55790b2b9258f03539387442f4b01eb0e1ed633a935f3f4da34e5fa8acecdbb3"} Oct 30 09:18:51.880265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.880158 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w"] Oct 30 09:18:51.972702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.972659 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.972917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.972776 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.973121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.973097 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pds\" (UniqueName: \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.974293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.974269 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:51.977883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:51.977845 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:52.029207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.029085 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pds\" (UniqueName: \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\") pod \"maas-api-revoke-sub-e874d9c8-w2p2w\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:52.042272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.040670 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:52.042272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.040729 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:52.087561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.087525 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:18:52.125252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.124354 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:52.125252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.124419 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:52.670825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.658231 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:52.711316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:52.711278 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:18:53.149826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.132926 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.149826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.132998 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:53.149826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.133721 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.149826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.133764 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:53.150722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.150684 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.150893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.150868 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:53.329559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.329515 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:53.343405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.340252 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podStartSLOduration=4.34023281 podStartE2EDuration="4.34023281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:51.948354413 +0000 UTC m=+1495.671781591" watchObservedRunningTime="2026-10-30 09:18:53.34023281 +0000 UTC m=+1497.063659951" Oct 30 09:18:53.352962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.352858 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w"] Oct 30 09:18:53.593289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.592598 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.593289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:53.592658 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:54.037239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.036670 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.037239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.036732 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:54.093618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.093581 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.093929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.093887 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:54.275238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.274430 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" event={"ID":"944b6d05-12c6-43ef-8dae-300bd0db8e08","Type":"ContainerStarted","Data":"18ba9606ddba5da57e790d73a809a71d891b61f2ad026b8a3458bdf69e9cdad9"} Oct 30 09:18:54.275238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.274495 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" event={"ID":"944b6d05-12c6-43ef-8dae-300bd0db8e08","Type":"ContainerStarted","Data":"a5384eab3bdf2aa7f7c30f62cc352bf070a0393e851bc32a649dfdaeae9a0e7e"} Oct 30 09:18:54.329241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.328452 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" podStartSLOduration=3.328432772 podStartE2EDuration="3.328432772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:54.316467781 +0000 UTC m=+1498.039894925" watchObservedRunningTime="2026-10-30 09:18:54.328432772 +0000 UTC m=+1498.051859916" Oct 30 09:18:54.569241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.568694 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:54.594669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.593212 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.594669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.593286 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:54.895336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.895290 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:18:54.952282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.952250 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.959413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.957453 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:18:54.994237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994188 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994265 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994291 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994425 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994499 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9m5\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994531 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994566 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994584 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:54.994682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:54.994599 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.058241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.055367 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.058241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.055427 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:55.094431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.094336 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.094431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.094402 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:55.095451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095420 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095464 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095494 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095546 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095577 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095601 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095736 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095818 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9m5\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095858 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.095998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.095907 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.096253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.096210 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.096485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.096456 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.096846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.096827 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.120262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.097444 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.120262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.105765 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.149546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.149474 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.149761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.149735 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.150696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.150663 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9m5\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.308104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.306866 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:18:55.436827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.435592 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp"] Oct 30 09:18:55.462576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.461949 4920 generic.go:356] "Generic (PLEG): container finished" podID="944b6d05-12c6-43ef-8dae-300bd0db8e08" containerID="18ba9606ddba5da57e790d73a809a71d891b61f2ad026b8a3458bdf69e9cdad9" exitCode=0 Oct 30 09:18:55.497053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.497006 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" event={"ID":"944b6d05-12c6-43ef-8dae-300bd0db8e08","Type":"ContainerDied","Data":"18ba9606ddba5da57e790d73a809a71d891b61f2ad026b8a3458bdf69e9cdad9"} Oct 30 09:18:55.497293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.497128 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.597071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.595813 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp"] Oct 30 09:18:55.597071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.596189 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.597071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.596251 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:55.617265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.615041 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pff2j\" (UniqueName: \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.617265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.615142 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.617265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.615291 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.781306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.760008 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pff2j\" (UniqueName: \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.781306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.760097 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.781306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.760248 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.791272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.791236 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.817618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.817553 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:55.925671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:55.890503 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pff2j\" (UniqueName: \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\") pod \"maas-api-revoke-sub-3741f808-8nrxp\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:56.051246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.039652 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:56.051246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.039712 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:56.145258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.111623 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 09:18:56.145258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.111682 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 09:18:56.185898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.185104 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:18:56.649271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.619438 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:56.649271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.619537 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:56.853087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.850642 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:56.853087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.850817 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:56.853087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:56.850994 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:57.092242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.076686 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.111451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.111401 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.112975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.111729 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 09:18:57.113256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.113231 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 09:18:57.294896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.276699 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.294896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.277049 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:18:57.315240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.310155 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:18:57.315240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.310446 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:18:57.375242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.366600 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.765255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.735841 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:18:57.847334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.834650 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh"] Oct 30 09:18:57.963647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.805877 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:57.963647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.901116 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:57.981322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:57.834528 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:18:58.165418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.165371 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Oct 30 09:18:58.166894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.166865 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh"] Oct 30 09:18:58.172433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.172401 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.236242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.202207 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:18:58.236242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.203632 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:18:58.236242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.204960 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.236762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.236709 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:18:58.288259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.282152 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:18:58.318606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.318247 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:18:58.318883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.318853 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:18:58.320122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.319388 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:18:58.374451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.374332 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:18:58.411244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.376292 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:18:58.446258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.420331 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:18:58.448685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.448646 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.452035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.452006 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:58.452185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.452070 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.535030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.534986 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.565264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.547874 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.622863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.622826 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.623129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.623112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2lp\" (UniqueName: \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.671258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.670197 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:58.671258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.670278 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:58.730719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.730663 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.734529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.734486 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.738349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.738313 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2lp\" (UniqueName: \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:58.749259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:58.744907 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:59.136677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.136640 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2lp\" (UniqueName: \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:59.208270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.208026 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.208451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.208419 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:18:59.249305 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:18:59.246121 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123aa87d_d965_43db_9189_4fd0308b3dcc.slice/crio-9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3 WatchSource:0}: Error finding container 9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3: Status 404 returned error can't find the container with id 9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3 Oct 30 09:18:59.249305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.246612 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c935a589-tjhjh\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:59.287932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.287883 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:59.288245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.288189 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.289611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.289584 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.292715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.292695 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:18:59.308751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.308728 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp"] Oct 30 09:18:59.309016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.309004 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:18:59.409242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.331167 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.410007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.336328 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:59.452266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.445296 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.452266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.433043 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:18:59.452266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.409615 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.456339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.456294 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.539275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.539235 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:59.539533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.539497 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:59.612273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.609634 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:18:59.612273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.609694 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:18:59.612273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.610876 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" event={"ID":"8dd705ed-cf80-41b7-a90b-152da20bef54","Type":"ContainerStarted","Data":"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1"} Oct 30 09:18:59.612273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.610937 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" event={"ID":"8dd705ed-cf80-41b7-a90b-152da20bef54","Type":"ContainerStarted","Data":"fdd9161720addb168ab9da49702a6bfd1f7ded414b907d929eb69e2a00ed6ecd"} Oct 30 09:18:59.861383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.713476 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:18:59.922252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:18:59.917047 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:00.117293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.116437 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:19:00.209402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.209302 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.209952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.209924 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:00.230288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.225073 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\" (UniqueName: \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\") pod \"944b6d05-12c6-43ef-8dae-300bd0db8e08\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " Oct 30 09:19:00.230288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.225150 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\") pod \"944b6d05-12c6-43ef-8dae-300bd0db8e08\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " Oct 30 09:19:00.230288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.225469 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\") pod \"944b6d05-12c6-43ef-8dae-300bd0db8e08\" (UID: \"944b6d05-12c6-43ef-8dae-300bd0db8e08\") " Oct 30 09:19:00.234587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.234556 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:19:00.236978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.236829 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:19:00.294271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.294211 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.315409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.308355 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:19:00.320374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.319560 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:19:00.713243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.702488 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca" pod "944b6d05-12c6-43ef-8dae-300bd0db8e08" (UID: "944b6d05-12c6-43ef-8dae-300bd0db8e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:00.742278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.728043 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca" pod "944b6d05-12c6-43ef-8dae-300bd0db8e08" (UID: "944b6d05-12c6-43ef-8dae-300bd0db8e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:00.780295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.745170 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds" pod "944b6d05-12c6-43ef-8dae-300bd0db8e08" (UID: "944b6d05-12c6-43ef-8dae-300bd0db8e08"). InnerVolumeSpecName "kube-api-access-45pds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:00.816476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.816437 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:00.820410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.820366 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:00.923439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.923379 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:00.923725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.923695 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:00.974501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.974413 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:00.974695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.974681 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45pds\" (UniqueName: \"kubernetes.io/projected/944b6d05-12c6-43ef-8dae-300bd0db8e08-kube-api-access-45pds\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:00.974795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:00.974783 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b6d05-12c6-43ef-8dae-300bd0db8e08-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:01.074238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.069719 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:19:01.074238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.072006 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:19:01.083246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.081989 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.137731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.135802 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.181741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.181680 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.201267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.195616 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:19:01.201267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.195686 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:19:01.247299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.242172 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.455242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.424486 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:01.455624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.455579 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:01.612077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.612043 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 09:19:01.612368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.612340 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 09:19:01.645580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.645504 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podStartSLOduration=7.645477815 podStartE2EDuration="7.645477815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:59.685697651 +0000 UTC m=+1503.409124824" watchObservedRunningTime="2026-10-30 09:19:01.645477815 +0000 UTC m=+1505.368904960" Oct 30 09:19:01.767890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.286530 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.896985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.896937 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:01.929423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.928250 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:01.929423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.928333 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:01.991511 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:19:01.991475 4920 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.382s" Oct 30 09:19:01.991775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.991758 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:19:01.992094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.992074 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:19:01.996368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:01.996344 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:19:02.058329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.052888 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:19:02.086663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.086629 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:19:02.093555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.093249 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" event={"ID":"944b6d05-12c6-43ef-8dae-300bd0db8e08","Type":"ContainerDied","Data":"a5384eab3bdf2aa7f7c30f62cc352bf070a0393e851bc32a649dfdaeae9a0e7e"} Oct 30 09:19:02.093555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.093347 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5384eab3bdf2aa7f7c30f62cc352bf070a0393e851bc32a649dfdaeae9a0e7e" Oct 30 09:19:02.174245 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:19:02.171650 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:19:02.192946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.192914 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w" Oct 30 09:19:02.290627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.290568 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:19:02.360242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.352794 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:02.360242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.352856 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:02.367271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.365868 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:19:02.369727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.368288 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:19:02.374860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.374805 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" event={"ID":"123aa87d-d965-43db-9189-4fd0308b3dcc","Type":"ContainerStarted","Data":"9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3"} Oct 30 09:19:02.414931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.414854 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:19:02.795416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.795387 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.852387 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.852446 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh"] Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.852495 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx"] Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.853375 4920 cpu_manager.go:414] "containerMap: removing container" podUID="944b6d05-12c6-43ef-8dae-300bd0db8e08" containerName="revoke-keys" Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.853395 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b6d05-12c6-43ef-8dae-300bd0db8e08" containerName="revoke-keys" Oct 30 09:19:02.855704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.853591 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b6d05-12c6-43ef-8dae-300bd0db8e08" containerName="revoke-keys" Oct 30 09:19:02.884406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.884373 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx"] Oct 30 09:19:02.885875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.885373 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:02.932802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.932271 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:02.932802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.932367 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87jx\" (UniqueName: \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:02.932802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.932424 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:02.993290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:02.993035 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:19:03.018261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.015744 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:19:03.046236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.034545 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.046236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.034855 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.046236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.034920 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87jx\" (UniqueName: \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.046236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.038839 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.066763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.066731 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.071243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.069802 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87jx\" (UniqueName: \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\") pod \"maas-api-revoke-sub-20caa23c-q5ntx\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.257378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.257301 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:19:03.257745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.257714 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:19:03.258385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.258366 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:19:03.288127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.288094 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:03.328240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.324562 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:03.328240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.324620 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:03.464630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.458956 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:03.464630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.461073 4920 generic.go:356] "Generic (PLEG): container finished" podID="123aa87d-d965-43db-9189-4fd0308b3dcc" containerID="d2087846ba74fcbb10537699f9fdb3be0407e6a0ba98fe51c03c987b23d953ca" exitCode=0 Oct 30 09:19:03.464630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.461171 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" event={"ID":"123aa87d-d965-43db-9189-4fd0308b3dcc","Type":"ContainerDied","Data":"d2087846ba74fcbb10537699f9fdb3be0407e6a0ba98fe51c03c987b23d953ca"} Oct 30 09:19:03.464803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.464653 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" event={"ID":"1f9d39d0-8483-490b-a713-2d6d27389a4e","Type":"ContainerStarted","Data":"cf2a2bc524a94967a69fbfc2bb3156a8fbb48164eb7173d9a4b738494aa2c08b"} Oct 30 09:19:03.464803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.464714 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" event={"ID":"1f9d39d0-8483-490b-a713-2d6d27389a4e","Type":"ContainerStarted","Data":"d71cefed7430590e1611daca14b854d961788caa3bfb763b8c67b60ae58bf7f1"} Oct 30 09:19:03.749927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.749860 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" podStartSLOduration=6.749826977 podStartE2EDuration="6.749826977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:03.627612577 +0000 UTC m=+1507.351039722" watchObservedRunningTime="2026-10-30 09:19:03.749826977 +0000 UTC m=+1507.473254124" Oct 30 09:19:03.750227 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:03.750195 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx"] Oct 30 09:19:03.773599 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:19:03.773519 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c9bbd6_4104_4a17_ae06_7227131fdff4.slice/crio-bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23 WatchSource:0}: Error finding container bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23: Status 404 returned error can't find the container with id bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23 Oct 30 09:19:04.313296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.309813 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:04.313296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.309880 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:04.480211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.480166 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" event={"ID":"11c9bbd6-4104-4a17-ae06-7227131fdff4","Type":"ContainerStarted","Data":"1b1b7d4c48119d5ce2d9d231c94096d5f134be3a3acb1d2e4f9fdd4c602f4878"} Oct 30 09:19:04.480760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.480265 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" event={"ID":"11c9bbd6-4104-4a17-ae06-7227131fdff4","Type":"ContainerStarted","Data":"bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23"} Oct 30 09:19:04.482830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.482802 4920 generic.go:356] "Generic (PLEG): container finished" podID="1f9d39d0-8483-490b-a713-2d6d27389a4e" containerID="cf2a2bc524a94967a69fbfc2bb3156a8fbb48164eb7173d9a4b738494aa2c08b" exitCode=0 Oct 30 09:19:04.482978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.482907 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" event={"ID":"1f9d39d0-8483-490b-a713-2d6d27389a4e","Type":"ContainerDied","Data":"cf2a2bc524a94967a69fbfc2bb3156a8fbb48164eb7173d9a4b738494aa2c08b"} Oct 30 09:19:04.550567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.542646 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" podStartSLOduration=2.542623812 podStartE2EDuration="2.542623812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:04.536624787 +0000 UTC m=+1508.260051930" watchObservedRunningTime="2026-10-30 09:19:04.542623812 +0000 UTC m=+1508.266050959" Oct 30 09:19:04.841284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.841250 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:19:04.911845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.911746 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\") pod \"123aa87d-d965-43db-9189-4fd0308b3dcc\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " Oct 30 09:19:04.912566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.912543 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\") pod \"123aa87d-d965-43db-9189-4fd0308b3dcc\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " Oct 30 09:19:04.922253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.916531 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\" (UniqueName: \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\") pod \"123aa87d-d965-43db-9189-4fd0308b3dcc\" (UID: \"123aa87d-d965-43db-9189-4fd0308b3dcc\") " Oct 30 09:19:04.923590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.923001 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca" pod "123aa87d-d965-43db-9189-4fd0308b3dcc" (UID: "123aa87d-d965-43db-9189-4fd0308b3dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:04.924528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.924505 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:04.930471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.930417 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j" pod "123aa87d-d965-43db-9189-4fd0308b3dcc" (UID: "123aa87d-d965-43db-9189-4fd0308b3dcc"). InnerVolumeSpecName "kube-api-access-pff2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:04.933072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:04.933037 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca" pod "123aa87d-d965-43db-9189-4fd0308b3dcc" (UID: "123aa87d-d965-43db-9189-4fd0308b3dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:05.032283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.032243 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123aa87d-d965-43db-9189-4fd0308b3dcc-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:05.032283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.032284 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pff2j\" (UniqueName: \"kubernetes.io/projected/123aa87d-d965-43db-9189-4fd0308b3dcc-kube-api-access-pff2j\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:05.230170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.229904 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:05.318251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.315340 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:05.318251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.315415 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:05.492185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.492100 4920 generic.go:356] "Generic (PLEG): container finished" podID="11c9bbd6-4104-4a17-ae06-7227131fdff4" containerID="1b1b7d4c48119d5ce2d9d231c94096d5f134be3a3acb1d2e4f9fdd4c602f4878" exitCode=0 Oct 30 09:19:05.492705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.492187 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" event={"ID":"11c9bbd6-4104-4a17-ae06-7227131fdff4","Type":"ContainerDied","Data":"1b1b7d4c48119d5ce2d9d231c94096d5f134be3a3acb1d2e4f9fdd4c602f4878"} Oct 30 09:19:05.493588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.493561 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" Oct 30 09:19:05.493651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.493596 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp" event={"ID":"123aa87d-d965-43db-9189-4fd0308b3dcc","Type":"ContainerDied","Data":"9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3"} Oct 30 09:19:05.493651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.493636 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9698fa4482c2e2c1daf95f2e0def977767f39378801ae484daecadce248b3ed3" Oct 30 09:19:05.736363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.736337 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:19:05.791620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.791586 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:19:05.792190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792175 4920 cpu_manager.go:414] "containerMap: removing container" podUID="123aa87d-d965-43db-9189-4fd0308b3dcc" containerName="revoke-keys" Oct 30 09:19:05.792265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792192 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123aa87d-d965-43db-9189-4fd0308b3dcc" containerName="revoke-keys" Oct 30 09:19:05.792265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792212 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1f9d39d0-8483-490b-a713-2d6d27389a4e" containerName="revoke-keys" Oct 30 09:19:05.792265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792232 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9d39d0-8483-490b-a713-2d6d27389a4e" containerName="revoke-keys" Oct 30 09:19:05.792367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792353 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9d39d0-8483-490b-a713-2d6d27389a4e" containerName="revoke-keys" Oct 30 09:19:05.792401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.792367 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="123aa87d-d965-43db-9189-4fd0308b3dcc" containerName="revoke-keys" Oct 30 09:19:05.796674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.796589 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.847785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.847580 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:19:05.848470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.848439 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\") pod \"1f9d39d0-8483-490b-a713-2d6d27389a4e\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " Oct 30 09:19:05.848691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.848670 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\") pod \"1f9d39d0-8483-490b-a713-2d6d27389a4e\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " Oct 30 09:19:05.848776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.848764 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\" (UniqueName: \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\") pod \"1f9d39d0-8483-490b-a713-2d6d27389a4e\" (UID: \"1f9d39d0-8483-490b-a713-2d6d27389a4e\") " Oct 30 09:19:05.849104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849009 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca" pod "1f9d39d0-8483-490b-a713-2d6d27389a4e" (UID: "1f9d39d0-8483-490b-a713-2d6d27389a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:05.849104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849029 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849114 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849177 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw567\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849268 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849347 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849377 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849408 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849512 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849543 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.849676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.849634 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:05.853169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.853127 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp" pod "1f9d39d0-8483-490b-a713-2d6d27389a4e" (UID: "1f9d39d0-8483-490b-a713-2d6d27389a4e"). InnerVolumeSpecName "kube-api-access-lt2lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:05.858800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.856698 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca" pod "1f9d39d0-8483-490b-a713-2d6d27389a4e" (UID: "1f9d39d0-8483-490b-a713-2d6d27389a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:05.950755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950714 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.950755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950759 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950778 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950843 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950858 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950879 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.950919 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951003 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw567\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951092 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951158 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9d39d0-8483-490b-a713-2d6d27389a4e-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:05.951382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951176 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2lp\" (UniqueName: \"kubernetes.io/projected/1f9d39d0-8483-490b-a713-2d6d27389a4e-kube-api-access-lt2lp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:05.951382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951308 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951308 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951530 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951597 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.951885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.951863 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.953309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.953287 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.953524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.953505 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.976540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.976504 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:05.976671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:05.976563 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw567\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:06.160513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.160427 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:06.335023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.334644 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:06.335023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.334716 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:06.389888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.389852 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f"] Oct 30 09:19:06.396557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.396526 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.441022 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.440992 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f"] Oct 30 09:19:06.462900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.462867 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.463088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.463057 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.463148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.463117 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwb7\" (UniqueName: \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.526168 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:19:06.526139 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ef2c48_92b4_4b4d_ace1_289e7b0ee771.slice/crio-6b7e5f2a6821003f9cb6abdff94327a8096d28cfae7e428b725cdaa874f1ca25 WatchSource:0}: Error finding container 6b7e5f2a6821003f9cb6abdff94327a8096d28cfae7e428b725cdaa874f1ca25: Status 404 returned error can't find the container with id 6b7e5f2a6821003f9cb6abdff94327a8096d28cfae7e428b725cdaa874f1ca25 Oct 30 09:19:06.527228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.527186 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:19:06.527824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.527804 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" Oct 30 09:19:06.527949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.527818 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh" event={"ID":"1f9d39d0-8483-490b-a713-2d6d27389a4e","Type":"ContainerDied","Data":"d71cefed7430590e1611daca14b854d961788caa3bfb763b8c67b60ae58bf7f1"} Oct 30 09:19:06.528051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.527971 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71cefed7430590e1611daca14b854d961788caa3bfb763b8c67b60ae58bf7f1" Oct 30 09:19:06.529624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.529586 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:06.529725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.529672 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:06.529725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.529716 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:06.565615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.564828 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.565615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.564932 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.565615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.564966 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwb7\" (UniqueName: \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.566053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.566035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.567806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.567786 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.588806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.588170 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwb7\" (UniqueName: \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\") pod \"maas-api-revoke-sub-96fa4640-7zf7f\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.694943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.694915 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:06.742028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.741992 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:06.874475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.873038 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\" (UniqueName: \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\") pod \"11c9bbd6-4104-4a17-ae06-7227131fdff4\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " Oct 30 09:19:06.874475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.873204 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\") pod \"11c9bbd6-4104-4a17-ae06-7227131fdff4\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " Oct 30 09:19:06.874475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.873302 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\") pod \"11c9bbd6-4104-4a17-ae06-7227131fdff4\" (UID: \"11c9bbd6-4104-4a17-ae06-7227131fdff4\") " Oct 30 09:19:06.874475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.874140 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca" pod "11c9bbd6-4104-4a17-ae06-7227131fdff4" (UID: "11c9bbd6-4104-4a17-ae06-7227131fdff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:06.874475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.874409 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:06.884990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.884935 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca" pod "11c9bbd6-4104-4a17-ae06-7227131fdff4" (UID: "11c9bbd6-4104-4a17-ae06-7227131fdff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:06.885340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.885309 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx" pod "11c9bbd6-4104-4a17-ae06-7227131fdff4" (UID: "11c9bbd6-4104-4a17-ae06-7227131fdff4"). InnerVolumeSpecName "kube-api-access-p87jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:06.976342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.976253 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87jx\" (UniqueName: \"kubernetes.io/projected/11c9bbd6-4104-4a17-ae06-7227131fdff4-kube-api-access-p87jx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:06.976342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:06.976293 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9bbd6-4104-4a17-ae06-7227131fdff4-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:07.122269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.118437 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f"] Oct 30 09:19:07.165131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.165078 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:07.316885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.315786 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 09:19:07.316885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.315849 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 09:19:07.560833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.560796 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" event={"ID":"36ef2c48-92b4-4b4d-ace1-289e7b0ee771","Type":"ContainerStarted","Data":"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610"} Oct 30 09:19:07.561292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.560891 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" event={"ID":"36ef2c48-92b4-4b4d-ace1-289e7b0ee771","Type":"ContainerStarted","Data":"6b7e5f2a6821003f9cb6abdff94327a8096d28cfae7e428b725cdaa874f1ca25"} Oct 30 09:19:07.562590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.562563 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" event={"ID":"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b","Type":"ContainerStarted","Data":"5fc544bf93e45796b634333752f83e62e7334f35adfbf912cd1c3aef38e849a7"} Oct 30 09:19:07.563955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.563933 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" Oct 30 09:19:07.564056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.563948 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx" event={"ID":"11c9bbd6-4104-4a17-ae06-7227131fdff4","Type":"ContainerDied","Data":"bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23"} Oct 30 09:19:07.564056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.563988 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb3d70004025b240ad836d7b287c7428eebe455aaee92da4a7fea359b1cfc23" Oct 30 09:19:07.634199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.634125 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podStartSLOduration=2.634102593 podStartE2EDuration="2.634102593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:07.619926051 +0000 UTC m=+1511.343353193" watchObservedRunningTime="2026-10-30 09:19:07.634102593 +0000 UTC m=+1511.357529737" Oct 30 09:19:07.718671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:07.718601 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" podStartSLOduration=1.718581441 podStartE2EDuration="1.718581441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:07.713707056 +0000 UTC m=+1511.437134200" watchObservedRunningTime="2026-10-30 09:19:07.718581441 +0000 UTC m=+1511.442008587" Oct 30 09:19:08.160773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.160734 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:08.162696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.162658 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:08.162848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.162726 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:08.330241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.319473 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:19:08.330241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.320813 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:19:08.333704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.333681 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:19:08.576681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.576651 4920 generic.go:356] "Generic (PLEG): container finished" podID="d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" containerID="fe13fd8945a838281e529c9109fc4a625c3f27b82ac3040531cd233fac6bdc6c" exitCode=0 Oct 30 09:19:08.577140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:08.576730 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" event={"ID":"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b","Type":"ContainerDied","Data":"fe13fd8945a838281e529c9109fc4a625c3f27b82ac3040531cd233fac6bdc6c"} Oct 30 09:19:09.161116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.161079 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:09.161315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.161136 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:09.233241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.232729 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:09.838142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.838066 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:09.943034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.942995 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\") pod \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " Oct 30 09:19:09.943185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.943158 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\" (UniqueName: \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\") pod \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " Oct 30 09:19:09.943412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.943391 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\") pod \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\" (UID: \"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b\") " Oct 30 09:19:09.944611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.944575 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca" pod "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" (UID: "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:09.946603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.946577 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7" pod "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" (UID: "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b"). InnerVolumeSpecName "kube-api-access-4hwb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:09.947321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:09.947289 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca" pod "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" (UID: "d7e04fa0-c4f4-4e25-8a83-c3172072bc9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:10.044621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.044584 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:10.044621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.044617 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:10.044621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.044629 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwb7\" (UniqueName: \"kubernetes.io/projected/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b-kube-api-access-4hwb7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:10.131329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.131228 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:10.161614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.161585 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:10.161783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.161636 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:10.641179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.641133 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" event={"ID":"d7e04fa0-c4f4-4e25-8a83-c3172072bc9b","Type":"ContainerDied","Data":"5fc544bf93e45796b634333752f83e62e7334f35adfbf912cd1c3aef38e849a7"} Oct 30 09:19:10.641444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.641202 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc544bf93e45796b634333752f83e62e7334f35adfbf912cd1c3aef38e849a7" Oct 30 09:19:10.641444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.641297 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.647346 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648045 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" containerName="revoke-keys" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648063 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" containerName="revoke-keys" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648109 4920 cpu_manager.go:414] "containerMap: removing container" podUID="11c9bbd6-4104-4a17-ae06-7227131fdff4" containerName="revoke-keys" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648114 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c9bbd6-4104-4a17-ae06-7227131fdff4" containerName="revoke-keys" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648203 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c9bbd6-4104-4a17-ae06-7227131fdff4" containerName="revoke-keys" Oct 30 09:19:10.651588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.648231 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" containerName="revoke-keys" Oct 30 09:19:10.663605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.663577 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:19:10.663756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.663687 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.675611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.674793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:10.687189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.687155 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8eada0cf-openshift-default-dockercfg-p27vt\"" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896593 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896665 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4g5c\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896720 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896776 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896907 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.896958 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.897010 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.897094 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.897248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.897133 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:10.986451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:10.986411 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:11.005639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.005583 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4g5c\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.006394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.006355 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.006628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.006609 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.006830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.006814 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.006945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.006931 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.007074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.007058 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.007272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.007253 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.007414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.007397 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.009625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.007612 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.023873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.010776 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.024183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.011559 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.024354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.011802 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.024454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.012683 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.024552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.013339 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.037745 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.038752 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.043243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.039004 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.044184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.044141 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4g5c\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\") pod \"e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.161646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.161565 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:11.161646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.161616 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:11.282519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.282483 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:11.674172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.674133 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:11.686685 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:19:11.686650 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6b2f06_9e0e_4f39_ba1d_f3e1c7a6f148.slice/crio-6254d3c4c8c97b10cc5bdba9150b027f883f0a36a22b103e4aae9d76effad092 WatchSource:0}: Error finding container 6254d3c4c8c97b10cc5bdba9150b027f883f0a36a22b103e4aae9d76effad092: Status 404 returned error can't find the container with id 6254d3c4c8c97b10cc5bdba9150b027f883f0a36a22b103e4aae9d76effad092 Oct 30 09:19:11.689801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.689767 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:11.689914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.689852 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:11.689914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.689897 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:11.690684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:11.690653 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:19:12.161560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:12.161525 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:12.161961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:12.161576 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:12.642720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:12.642679 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:12.665527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:12.665480 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerStarted","Data":"4a7776718e62660ed5cbd6010a9d57c37b3e55d80779ed0bf8cc38e3e55a1bd8"} Oct 30 09:19:12.665705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:12.665556 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerStarted","Data":"6254d3c4c8c97b10cc5bdba9150b027f883f0a36a22b103e4aae9d76effad092"} Oct 30 09:19:13.161028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.160991 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 09:19:13.161210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.161050 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 09:19:13.283725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.283681 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:13.290631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.290596 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:13.348184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.348127 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podStartSLOduration=3.348112388 podStartE2EDuration="3.348112388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:12.747761514 +0000 UTC m=+1516.471188694" watchObservedRunningTime="2026-10-30 09:19:13.348112388 +0000 UTC m=+1517.071539530" Oct 30 09:19:13.537709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.537666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:13.678201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.678166 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:13.679469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:13.679446 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:19:14.165331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:14.165300 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:14.165730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:14.165708 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:14.166464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:14.166447 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:19:15.308971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.308933 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:15.455965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.455930 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:15.806351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.806313 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v"] Oct 30 09:19:15.811832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.811807 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:15.848232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.848185 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v"] Oct 30 09:19:15.851632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.851607 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:19:15.851819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.851794 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:19:15.851819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.851605 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:19:15.851965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.851889 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:19:15.929600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.929562 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9k2\" (UniqueName: \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:15.929768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.929620 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:15.929768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:15.929719 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.031634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.031583 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9k2\" (UniqueName: \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.031818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.031649 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.031911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.031817 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.032667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.032636 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.034259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.034229 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.067495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.067420 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9k2\" (UniqueName: \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\") pod \"maas-api-revoke-sub-129e40f5-4qm5v\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.127779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.127740 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:16.358769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.358669 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:16.453695 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:19:16.453660 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d4dd57_2eb7_45f1_b53d_ff9d2619b164.slice/crio-a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b WatchSource:0}: Error finding container a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b: Status 404 returned error can't find the container with id a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b Oct 30 09:19:16.456367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.456335 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v"] Oct 30 09:19:16.536975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.536936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:16.706592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:16.706544 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" event={"ID":"70d4dd57-2eb7-45f1-b53d-ff9d2619b164","Type":"ContainerStarted","Data":"a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b"} Oct 30 09:19:17.384598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:17.376087 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:17.616471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:17.616438 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:17.725960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:17.719758 4920 generic.go:356] "Generic (PLEG): container finished" podID="70d4dd57-2eb7-45f1-b53d-ff9d2619b164" containerID="91d4734f30aa9b9080df9b5e3fcfde07cc4a078ad628c77da8a9fea2475c0568" exitCode=0 Oct 30 09:19:17.725960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:17.720067 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" event={"ID":"70d4dd57-2eb7-45f1-b53d-ff9d2619b164","Type":"ContainerDied","Data":"91d4734f30aa9b9080df9b5e3fcfde07cc4a078ad628c77da8a9fea2475c0568"} Oct 30 09:19:17.903275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:17.902317 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:18.139233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.139179 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:18.447046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.446953 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:19:18.461482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.461449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:18.529549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.529519 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:19:18.534379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.534353 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.588204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.588160 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:19:18.591098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.591069 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-648892-openshift-default-dockercfg-swzpr\"" Oct 30 09:19:18.594817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.594790 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.594944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.594846 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595026 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595069 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595095 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595249 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsdq\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595345 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595433 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.595614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.595597 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.727761 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.727851 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.727899 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.727966 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728031 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728058 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.728309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728192 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsdq\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728872 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728918 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.728947 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.729179 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.730019 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.730297 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.735250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.730533 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.843904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.843862 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.844329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.843869 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.844480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.844072 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.844939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.844876 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:18.847547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.847522 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsdq\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\") pod \"e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:18.854595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:18.854085 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:19.030271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.026949 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:19.085305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.085272 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:19.181881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.180360 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\") pod \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " Oct 30 09:19:19.181881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.180477 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\") pod \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " Oct 30 09:19:19.181881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.180634 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\" (UniqueName: \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\") pod \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\" (UID: \"70d4dd57-2eb7-45f1-b53d-ff9d2619b164\") " Oct 30 09:19:19.239240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.239124 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca" pod "70d4dd57-2eb7-45f1-b53d-ff9d2619b164" (UID: "70d4dd57-2eb7-45f1-b53d-ff9d2619b164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:19.243318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.239591 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca" pod "70d4dd57-2eb7-45f1-b53d-ff9d2619b164" (UID: "70d4dd57-2eb7-45f1-b53d-ff9d2619b164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:19.243318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.242013 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2" pod "70d4dd57-2eb7-45f1-b53d-ff9d2619b164" (UID: "70d4dd57-2eb7-45f1-b53d-ff9d2619b164"). InnerVolumeSpecName "kube-api-access-wg9k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.296820 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.296852 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9k2\" (UniqueName: \"kubernetes.io/projected/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-kube-api-access-wg9k2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.296867 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d4dd57-2eb7-45f1-b53d-ff9d2619b164-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.298988 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.299081 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:19.307347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.299128 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:19.312415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.312380 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:19:19.494896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.494804 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4"] Oct 30 09:19:19.496381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.496353 4920 cpu_manager.go:414] "containerMap: removing container" podUID="70d4dd57-2eb7-45f1-b53d-ff9d2619b164" containerName="revoke-keys" Oct 30 09:19:19.496530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.496514 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d4dd57-2eb7-45f1-b53d-ff9d2619b164" containerName="revoke-keys" Oct 30 09:19:19.497390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.497368 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d4dd57-2eb7-45f1-b53d-ff9d2619b164" containerName="revoke-keys" Oct 30 09:19:19.502743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.501965 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.525394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.525361 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4"] Oct 30 09:19:19.601738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.601693 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhv8f\" (UniqueName: \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.601738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.601751 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.602011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.601970 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.717240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.711291 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.717240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.711349 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhv8f\" (UniqueName: \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.717240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.711387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.741239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.722585 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.749508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.749380 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.770424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.753173 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhv8f\" (UniqueName: \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\") pod \"maas-api-revoke-sub-dd08f3e8-zhhk4\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.834239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.829514 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:19.983241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.973872 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" event={"ID":"70d4dd57-2eb7-45f1-b53d-ff9d2619b164","Type":"ContainerDied","Data":"a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b"} Oct 30 09:19:19.984375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.984350 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4472770c6c14103c1c3db0ebdbbad1a0e3f07f4917164811cb247f4a3cf558b" Oct 30 09:19:19.984553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:19.983542 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v" Oct 30 09:19:20.018333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.006681 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerStarted","Data":"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4"} Oct 30 09:19:20.018333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.006748 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerStarted","Data":"48baf9a61f1f971d6210581a6dbb917a97cde4894370588b78534473aa8f52a9"} Oct 30 09:19:20.090250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.072420 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podStartSLOduration=2.072292065 podStartE2EDuration="2.072292065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:20.071788208 +0000 UTC m=+1523.795215377" watchObservedRunningTime="2026-10-30 09:19:20.072292065 +0000 UTC m=+1523.795719210" Oct 30 09:19:20.841291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.830427 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:20.990654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.964297 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:20.990654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.968415 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:20.998421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:20.997909 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:21.118288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.093309 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:19:21.162721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.147636 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.188875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.187520 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:21.202701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.201523 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:19:21.216438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.215333 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.341656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.341601 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.369461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.369408 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:19:21.498391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.498344 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.705264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.704289 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:19:21.705264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.704759 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.783590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.783543 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.842186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.833127 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.845525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.842974 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.858773 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.858851 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.858911 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mg6\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.858971 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.858996 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.859046 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.859090 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.859162 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.862240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.859398 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965188 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965277 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965330 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965356 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mg6\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965416 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965440 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965502 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965548 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:21.998306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:21.965631 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.121324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.075984 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.122458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.089627 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.122631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.089875 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.122743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.102614 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.122834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.122314 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.123952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.123927 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:19:22.131240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.127423 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.153497 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.153781 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.154460 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.154918 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mg6\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.158762 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.201049 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4"] Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.221033 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:19:22.243251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.221689 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:19:22.417685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.417625 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.418731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.418701 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:19:22.593244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:22.591732 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:23.038727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.033394 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:23.135242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.131589 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.173764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.173668 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:23.229480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.209536 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:23.231707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.231651 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.290251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.263565 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.290251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.264062 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:19:23.339244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.309515 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.391263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.390134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:23.592381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.592335 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.667519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.667449 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.806476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.806421 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:19:23.820443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.820398 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" event={"ID":"dfc28ea1-392d-4404-afe1-5158ad5c8f5d","Type":"ContainerStarted","Data":"541c89cb9be9dbaa32a47245ca2c6d028f9d6071c6d494d5b1a6d8528039a0f2"} Oct 30 09:19:23.834378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:23.825410 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:19:24.150507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.150404 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.242577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.242532 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.490295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.242655 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.494240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.325753 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:19:24.507253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.325802 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.514263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.340333 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.518165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.518123 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:24.518568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.518426 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.719448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.719385 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.812538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:24.812470 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:25.166300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.163425 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:25.188242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.184901 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:25.202244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.198536 4920 generic.go:356] "Generic (PLEG): container finished" podID="dfc28ea1-392d-4404-afe1-5158ad5c8f5d" containerID="c9f51f560bcd007d951f8ab9c7fb3e77c0f289546b4f5026a954350d51e79f8c" exitCode=0 Oct 30 09:19:25.338080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.328030 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:25.487872 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:19:25.487784 4920 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.09s" Oct 30 09:19:25.494431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.494385 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" event={"ID":"dfc28ea1-392d-4404-afe1-5158ad5c8f5d","Type":"ContainerDied","Data":"c9f51f560bcd007d951f8ab9c7fb3e77c0f289546b4f5026a954350d51e79f8c"} Oct 30 09:19:25.527466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.527404 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:25.590240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.565321 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:25.590240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.565748 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:25.590240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.566018 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:19:25.611971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.598311 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:25.820503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.762692 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:25.820503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.763783 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:25.820503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:25.814666 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:26.164439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.164320 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:26.204274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.189371 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:26.595312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.573032 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:19:26.595312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.592529 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:26.595312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.593356 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:19:26.613127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.613073 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:26.689394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.683719 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:26.689394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.684549 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:19:26.822444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.822381 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:26.822886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.822838 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:26.823098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.823078 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:26.831983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.831951 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:26.951094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:26.948570 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:19:27.087910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.066533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:27.161958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.158544 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:19:27.196503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.196459 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:27.461109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.426944 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:27.495869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.486866 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:19:27.512895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.502114 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:19:27.518547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.518520 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:19:27.518870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.518854 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:19:27.518990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.518981 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:19:27.519102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.519093 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:19:27.519316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.519284 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.519537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.519285 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:19:27.519623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.519579 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:19:27.552798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.551571 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ca0df2-openshift-default-dockercfg-b2jlf\"" Oct 30 09:19:27.556351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.556314 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" event={"ID":"ff478156-3c2c-4a23-af6d-05771a5ccb17","Type":"ContainerStarted","Data":"c4136d18b874fdfdb182bb166e1b89c40d6148c6ee0785a0bcc45dfccad7608e"} Oct 30 09:19:27.609241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.608096 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652164 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652234 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652283 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652385 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652452 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.652585 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.653047 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.653099 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxch\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.653295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.653126 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.655582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.655540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:27.754524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.754431 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\") pod \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " Oct 30 09:19:27.754855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.754823 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\" (UniqueName: \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\") pod \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " Oct 30 09:19:27.755499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.755477 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca" pod "dfc28ea1-392d-4404-afe1-5158ad5c8f5d" (UID: "dfc28ea1-392d-4404-afe1-5158ad5c8f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:27.755669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.755650 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\") pod \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\" (UID: \"dfc28ea1-392d-4404-afe1-5158ad5c8f5d\") " Oct 30 09:19:27.756196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.756173 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.757021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.756997 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.758016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.757708 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.758016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.757955 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.758186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.758030 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxch\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.758186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.758074 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.759008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.758981 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.759352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.759325 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.759459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.759438 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.759631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.759608 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.759868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.759846 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.760126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.760107 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.760300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.760133 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.760412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.759889 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.760517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.760304 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.760617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.760437 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f" pod "dfc28ea1-392d-4404-afe1-5158ad5c8f5d" (UID: "dfc28ea1-392d-4404-afe1-5158ad5c8f5d"). InnerVolumeSpecName "kube-api-access-qhv8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:27.762409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.762386 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.762874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.762851 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.765567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.765536 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca" pod "dfc28ea1-392d-4404-afe1-5158ad5c8f5d" (UID: "dfc28ea1-392d-4404-afe1-5158ad5c8f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:19:27.775573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.775540 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.777311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.777283 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxch\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\") pod \"e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:27.862036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.861996 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.862036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.862037 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhv8f\" (UniqueName: \"kubernetes.io/projected/dfc28ea1-392d-4404-afe1-5158ad5c8f5d-kube-api-access-qhv8f\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:19:27.904904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:27.904866 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:28.294245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.292421 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:28.294245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.292551 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:28.294245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.292608 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:28.303938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.297426 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:19:28.568791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.568752 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" event={"ID":"ff478156-3c2c-4a23-af6d-05771a5ccb17","Type":"ContainerStarted","Data":"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9"} Oct 30 09:19:28.571339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.571311 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerStarted","Data":"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33"} Oct 30 09:19:28.571541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.571516 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerStarted","Data":"11fc738d917699f1646ed39c2edcfee9fa71e7ae81e9e240a70d81cf5aae9217"} Oct 30 09:19:28.572923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.572903 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" event={"ID":"dfc28ea1-392d-4404-afe1-5158ad5c8f5d","Type":"ContainerDied","Data":"541c89cb9be9dbaa32a47245ca2c6d028f9d6071c6d494d5b1a6d8528039a0f2"} Oct 30 09:19:28.572923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.572919 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4" Oct 30 09:19:28.573056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.572944 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541c89cb9be9dbaa32a47245ca2c6d028f9d6071c6d494d5b1a6d8528039a0f2" Oct 30 09:19:28.645697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.645639 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podStartSLOduration=8.645620811 podStartE2EDuration="8.645620811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:28.619811641 +0000 UTC m=+1532.343238785" watchObservedRunningTime="2026-10-30 09:19:28.645620811 +0000 UTC m=+1532.369047953" Oct 30 09:19:28.691867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.691808 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podStartSLOduration=2.691793203 podStartE2EDuration="2.691793203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:28.691622879 +0000 UTC m=+1532.415050041" watchObservedRunningTime="2026-10-30 09:19:28.691793203 +0000 UTC m=+1532.415220346" Oct 30 09:19:28.916540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:28.905314 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:29.235017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.234925 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:29.236127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.236104 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:29.236245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.236151 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:29.291555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.291518 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:29.519299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.511643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:29.626789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.622941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:29.791647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.791614 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:29.926646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:29.926601 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:30.234437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:30.234354 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:30.234437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:30.234406 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:30.594042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:30.594006 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:30.595020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:30.594998 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:19:31.071959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.071925 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:31.151816 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:19:31.151779 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:19:31.176538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.176503 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:31.234695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.234654 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:31.234875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.234713 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:31.372959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.372871 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:31.497592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.497556 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 09:19:31.498649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.498615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:31.499001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.498925 4920 cpu_manager.go:414] "containerMap: removing container" podUID="dfc28ea1-392d-4404-afe1-5158ad5c8f5d" containerName="revoke-keys" Oct 30 09:19:31.499001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.498952 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc28ea1-392d-4404-afe1-5158ad5c8f5d" containerName="revoke-keys" Oct 30 09:19:31.499158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.499149 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc28ea1-392d-4404-afe1-5158ad5c8f5d" containerName="revoke-keys" Oct 30 09:19:31.530472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.530442 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.593412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.593368 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 09:19:31.602352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.594885 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h44vk\"" Oct 30 09:19:31.607620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.595148 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 09:19:31.607620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.595188 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 09:19:31.643889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.643785 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 09:19:31.663206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663168 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663245 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chb62\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663297 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663343 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663399 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663426 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.663498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.663458 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.764849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.764799 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.764871 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chb62\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.764947 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.765025 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.765096 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.765128 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.765164 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.765853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.765827 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.768032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.768006 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.768132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.768011 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.768132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.768059 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.768132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.768086 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.786399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.786364 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.787666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.787634 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chb62\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97vfn\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:31.924290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:31.923888 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:32.238863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.235698 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:32.238863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.235768 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:32.264505 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:19:32.263425 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccda3a0_e93c_4d5b_a694_51e310cab997.slice/crio-6e40db0a67c244057482d41c1e2ad828eeacd2f4cea71f9fdd41ae26f8ec6ea4 WatchSource:0}: Error finding container 6e40db0a67c244057482d41c1e2ad828eeacd2f4cea71f9fdd41ae26f8ec6ea4: Status 404 returned error can't find the container with id 6e40db0a67c244057482d41c1e2ad828eeacd2f4cea71f9fdd41ae26f8ec6ea4 Oct 30 09:19:32.267549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.267522 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:19:32.273416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.273392 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 09:19:32.671406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.671371 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" event={"ID":"2ccda3a0-e93c-4d5b-a694-51e310cab997","Type":"ContainerStarted","Data":"6e40db0a67c244057482d41c1e2ad828eeacd2f4cea71f9fdd41ae26f8ec6ea4"} Oct 30 09:19:32.865274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.865236 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:32.962526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:32.962435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:33.260240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:33.251338 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:33.260240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:33.251406 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:33.362316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:33.362276 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:34.234846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:34.234811 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 09:19:34.235035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:34.234888 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 09:19:35.097590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.097526 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:35.712557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.712523 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:35.714005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.713980 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:35.714460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.714440 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:19:35.741491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.741455 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:35.741601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:35.741548 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:36.720409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:36.720373 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" event={"ID":"2ccda3a0-e93c-4d5b-a694-51e310cab997","Type":"ContainerStarted","Data":"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb"} Oct 30 09:19:36.721024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:36.720615 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:36.722345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:36.722319 4920 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-97vfn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:19:36.722480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:36.722364 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" podUID="2ccda3a0-e93c-4d5b-a694-51e310cab997" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:19:36.797962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:36.797926 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:37.017804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:37.017713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:37.136624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:37.136589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:37.234948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:37.234917 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:37.727892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:37.727860 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 09:19:37.802533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:37.802472 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" podStartSLOduration=3.330432494 podStartE2EDuration="6.802455493s" totalImagesPullingTime="3.472022999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:36.794717058 +0000 UTC m=+1540.518144222" watchObservedRunningTime="2026-10-30 09:19:37.802455493 +0000 UTC m=+1541.525882635" Oct 30 09:19:39.083705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:39.083674 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:40.245313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.245263 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:19:40.251154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.251129 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.290090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.290054 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:19:40.385719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.385682 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.385896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.385806 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.385896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.385858 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.385973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.385898 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.386030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.386013 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qf9\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.386066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.386046 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.386100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.386064 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.386269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.386208 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.386315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.386295 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487469 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487519 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qf9\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487561 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487583 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487669 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487692 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487806 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487907 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.487987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.487922 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.488111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.488019 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.488111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.488066 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.488111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.488075 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.488372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.488353 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.488606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.488584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.490278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.490256 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.490499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.490482 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.508282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.508129 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qf9\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.508433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.508289 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.574347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.574313 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:40.879694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.879615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:40.906722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.906686 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:40.906839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.906765 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:40.906839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.906832 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:40.918756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:40.910398 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:19:41.766574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:41.766531 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" event={"ID":"e38f1e38-8109-41b7-84a1-8b83fa173850","Type":"ContainerStarted","Data":"bce1e3add6476bb7b78ef2fcca2c1253c05f11540b14843180aa326b1899d620"} Oct 30 09:19:41.767083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:41.766593 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" event={"ID":"e38f1e38-8109-41b7-84a1-8b83fa173850","Type":"ContainerStarted","Data":"c4c96db62f5a495c167f92ea292f9f9ba7ce1b01d7cbb5f0a6c1a21d449723a0"} Oct 30 09:19:41.845385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:41.845327 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podStartSLOduration=1.845312239 podStartE2EDuration="1.845312239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:41.828692215 +0000 UTC m=+1545.552119359" watchObservedRunningTime="2026-10-30 09:19:41.845312239 +0000 UTC m=+1545.568739382" Oct 30 09:19:42.574686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:42.574636 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:42.576505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:42.576477 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:42.576654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:42.576536 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:42.615241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:42.606693 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:43.575021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:43.574983 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:43.575489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:43.575034 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:44.380842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:44.380810 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:44.575603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:44.575566 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:44.576004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:44.575618 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:45.575127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:45.575090 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:45.575337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:45.575165 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:46.576236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:46.575631 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:46.576236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:46.575732 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:47.142277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.126824 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:47.430670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.416678 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:47.520265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.519356 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.528673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.528622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:47.613258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.602201 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:47.613258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.602454 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:47.653246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.629952 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.773299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:47.772307 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:48.039247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.037311 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.319308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.318302 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 09:19:48.421244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.414137 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:48.529242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.519387 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.602628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.574334 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.602628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.574712 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 09:19:48.602628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.598984 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:48.602628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.601072 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:48.602628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.601113 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:48.646347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.635568 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:19:48.764253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.760208 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:19:48.764253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.760296 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:19:48.865301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.864875 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.904627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:48.904590 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:19:49.173294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.073080 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:49.346398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.346351 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.347486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.347461 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:19:49.416413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.416370 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.418715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.418670 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:19:49.567245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.556688 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.567245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.559628 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:19:49.591243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.590505 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:19:49.618243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.618004 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.668244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.621447 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:19:49.668244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.636338 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:49.668244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.636398 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:49.668244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.636954 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.886299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:49.885450 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:50.131362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.131318 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:50.152302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.144297 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.376423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.376374 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.417402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.417299 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.570034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.569327 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:19:50.882521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.880394 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.908725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.882132 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:50.911058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.882266 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.911863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.906733 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:50.989243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:50.988474 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.060164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.060128 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:51.060524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.060471 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:51.307237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.307178 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:51.320356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.320301 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.520902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.520861 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:51.522434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.522398 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.540251 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:19:51.540197 4920 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.139s" Oct 30 09:19:51.570441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.547892 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:51.570824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.569698 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.588104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.588066 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:51.588461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.588426 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:51.604242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.603301 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:19:51.717510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.717472 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:19:51.787381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.724425 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:51.788507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.749565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:51.910444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.910341 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:19:51.938917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.911847 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.977395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.911891 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.978484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.938299 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:52.003931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:51.951155 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.032822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.032786 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.033126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.033104 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.033312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.033290 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.037240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.033600 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khm4r\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.037466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.037443 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.041254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.037640 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.041494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.041470 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.066376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.066336 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.618443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.607930 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.679272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.667313 4920 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9mxzf container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:52.679272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.667389 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mxzf" podUID="adbbc457-38a1-45fd-ab3d-738f7dd07913" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727632 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727693 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727746 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727778 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727899 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khm4r\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.727942 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.728031 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.728078 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.728517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.728126 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.729200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.728805 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.823385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.798708 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.823385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.817193 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.823385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.821120 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.823385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.821972 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.833836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.828520 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 09:19:52.833836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.828576 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 09:19:52.848273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.834259 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:52.848273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.834940 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.848273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.802667 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.873325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.853073 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.873325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.857035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khm4r\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.944119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.944077 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:52.981793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.944751 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.984633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.984574 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:52.987059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:52.984921 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:53.020886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.017786 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:53.125420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.122641 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:19:53.141740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.140663 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:19:53.141740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.140905 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" containerID="cri-o://9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45" gracePeriod=30 Oct 30 09:19:53.157706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.149025 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:19:53.164710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.162629 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:19:53.328774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.328463 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:53.343250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.335934 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:53.624017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.623984 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 09:19:53.848240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.847897 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:53.875295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.874786 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:53.991370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:53.991317 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:54.028916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.028863 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:19:54.655259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.654834 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 09:19:54.655779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.655755 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:19:54.701327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.683627 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:54.701327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.683688 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:54.871254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.848460 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:19:54.994210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:54.994150 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:19:55.063276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.060319 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:55.220539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.220491 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:55.249831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.249798 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:19:55.299084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.299037 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:55.299410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.299371 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:55.299614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.299588 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:55.431817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.431784 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:19:55.474203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.474175 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:19:55.588716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.588684 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:55.766859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.766423 4920 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:55.766859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.766488 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:55.901789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:55.901752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:56.368860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.368549 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" event={"ID":"14134030-ed0b-46f1-bdbb-4d5a9edb4966","Type":"ContainerStarted","Data":"b67e544987bfb5044be6f58e704b4a2606a3c4316030a9be27dcc13efe23bcc3"} Oct 30 09:19:56.368860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.368631 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" event={"ID":"14134030-ed0b-46f1-bdbb-4d5a9edb4966","Type":"ContainerStarted","Data":"db8209cc69a5e5676ce9e7b61f0c2c15e8eb276808b08765dfc67f0f06e6a387"} Oct 30 09:19:56.464256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.461597 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podStartSLOduration=7.461580081 podStartE2EDuration="7.461580081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:56.436584865 +0000 UTC m=+1560.160012036" watchObservedRunningTime="2026-10-30 09:19:56.461580081 +0000 UTC m=+1560.185007226" Oct 30 09:19:56.483792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.483758 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:56.485458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.485431 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:19:56.870940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:56.870905 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:57.341244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:57.329884 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:19:57.357047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:57.357011 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:19:57.357353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:57.357320 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:19:57.730335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:57.727443 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:58.330645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:58.330127 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:19:58.330645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:58.330195 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:19:58.498788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:58.498708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:19:58.823364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:58.823323 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kg5zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:19:58.823881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:58.823390 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:19:59.270276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.270190 4920 scope.go:122] "RemoveContainer" containerID="faff55648fc419c09d2e9c6122e23a95b49c15987b5e07d42f58ac7cae20997a" Oct 30 09:19:59.284073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.284046 4920 scope.go:122] "RemoveContainer" containerID="7827df439def7acf8046e09c86873986ec51f69d4561446696dab8f874776965" Oct 30 09:19:59.295411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.295389 4920 scope.go:122] "RemoveContainer" containerID="4fb8c18303603dda2da4c047e5491047686a46797ddd1ca94079c86704e701b2" Oct 30 09:19:59.329625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.329576 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:19:59.329793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.329652 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:19:59.345412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.345370 4920 scope.go:122] "RemoveContainer" containerID="3cb5f9623f7180b83f02395190e436e7ecf0c9d5f00d1c97776f638a4fa0c27d" Oct 30 09:19:59.356024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.356002 4920 scope.go:122] "RemoveContainer" containerID="6d34a677a217e484418e388fa00136ccad8bd9bfb01f69d86e64e9b4c379dc7b" Oct 30 09:19:59.722242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.721323 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:59.722242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:19:59.721403 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:20:00.150434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.150406 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:00.338269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.335532 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:20:00.338269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.335589 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:20:00.466767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.466189 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:20:00.476353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.476318 4920 generic.go:356] "Generic (PLEG): container finished" podID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerID="9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45" exitCode=0 Oct 30 09:20:00.476551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.476377 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" event={"ID":"3660829a-32df-4c3f-bbe7-af4ce2930250","Type":"ContainerDied","Data":"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45"} Oct 30 09:20:00.476551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.476392 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" Oct 30 09:20:00.476551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.476430 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv" event={"ID":"3660829a-32df-4c3f-bbe7-af4ce2930250","Type":"ContainerDied","Data":"880c88a1d031d41b0c53f850927d4bc59d830868df587b5343216d0de526ac97"} Oct 30 09:20:00.476551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.476459 4920 scope.go:122] "RemoveContainer" containerID="9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45" Oct 30 09:20:00.488691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.488661 4920 scope.go:122] "RemoveContainer" containerID="9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45" Oct 30 09:20:00.489059 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:20:00.489031 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45\": container with ID starting with 9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45 not found: ID does not exist" containerID="9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45" Oct 30 09:20:00.489129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.489071 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45"} err="failed to get container status \"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45\": rpc error: code = NotFound desc = could not find container \"9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45\": container with ID starting with 9306ad8eaeeafb7ad4390d6b2654848a7f5b68c477ee840f5a064a2a59adbb45 not found: ID does not exist" Oct 30 09:20:00.568064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568029 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568075 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568177 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568200 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568285 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568320 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568374 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.568435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568416 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.569668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.568453 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\") pod \"3660829a-32df-4c3f-bbe7-af4ce2930250\" (UID: \"3660829a-32df-4c3f-bbe7-af4ce2930250\") " Oct 30 09:20:00.569668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.569444 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:20:00.569956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.569904 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:20:00.570442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.570136 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:20:00.570942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.570922 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:20:00.571869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.571089 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:20:00.577458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.577419 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:20:00.578402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.578370 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:20:00.580701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.580679 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:20:00.582265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.581085 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz" pod "3660829a-32df-4c3f-bbe7-af4ce2930250" (UID: "3660829a-32df-4c3f-bbe7-af4ce2930250"). InnerVolumeSpecName "kube-api-access-pzkgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669865 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669899 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660829a-32df-4c3f-bbe7-af4ce2930250-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669909 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669917 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzkgz\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-kube-api-access-pzkgz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669927 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669935 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669945 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669954 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.670632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.669962 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660829a-32df-4c3f-bbe7-af4ce2930250-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:00.837478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.837443 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:20:00.846295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:00.846245 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kg5zv"] Oct 30 09:20:01.153787 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:20:01.153751 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:20:01.341743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:01.340291 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:20:01.341743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:01.340347 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:20:01.755231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:01.755183 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:02.328967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:02.328929 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:20:02.329402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:02.328993 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:20:02.404688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:02.404641 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" path="/var/lib/kubelet/pods/3660829a-32df-4c3f-bbe7-af4ce2930250/volumes" Oct 30 09:20:02.781696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:02.781592 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:03.176079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.175991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:03.282939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.282906 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:03.329850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.329815 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 09:20:03.330294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.329885 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 09:20:03.380603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.380566 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:03.515344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.515251 4920 generic.go:356] "Generic (PLEG): container finished" podID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerID="4a7776718e62660ed5cbd6010a9d57c37b3e55d80779ed0bf8cc38e3e55a1bd8" exitCode=0 Oct 30 09:20:03.515495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.515332 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerDied","Data":"4a7776718e62660ed5cbd6010a9d57c37b3e55d80779ed0bf8cc38e3e55a1bd8"} Oct 30 09:20:03.515860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.515842 4920 scope.go:122] "RemoveContainer" containerID="4a7776718e62660ed5cbd6010a9d57c37b3e55d80779ed0bf8cc38e3e55a1bd8" Oct 30 09:20:03.517607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.517573 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:03.517733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.517650 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:03.517733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.517700 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:03.878568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:03.878529 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:04.334190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.334156 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:20:04.334730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.334707 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:20:04.335542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.335520 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:20:04.459674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.459637 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:04.719295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.718389 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:04.753240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:04.752654 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerStarted","Data":"dd0302aae6644f036606239568757ab33bde86992d40d6eefff68bd475a1265d"} Oct 30 09:20:05.218240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:05.211983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:05.283594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:05.283558 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:20:05.283809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:05.283786 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 09:20:05.283874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:05.283853 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 09:20:06.283822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.283787 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 09:20:06.284366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.283861 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 09:20:06.287595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.287567 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:20:06.290257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.288434 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" Oct 30 09:20:06.290257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.288457 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" Oct 30 09:20:06.290257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.288671 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3660829a-32df-4c3f-bbe7-af4ce2930250" containerName="istio-proxy" Oct 30 09:20:06.318242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.309692 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.458604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.458564 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:20:06.551741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.551696 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkj2s\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.551932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.551807 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.551996 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552038 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552249 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552325 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552437 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552477 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.552580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.552564 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657107 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657183 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657239 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657392 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657434 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657491 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657518 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657556 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.657684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.657663 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkj2s\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.661655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.661619 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.668194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.662532 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.668194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.662776 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.668194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.663000 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.668194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.663599 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.674244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.672662 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.717242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.716308 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkj2s\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.738246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.735869 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.738246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.736377 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-8k2jx\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:06.766242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:06.760534 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:07.025675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.023793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:07.227720 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:20:07.227687 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c83ecf_2406_4b6f_9b0d_00bf162a3e19.slice/crio-7bb8905c7f552cd9abe715354c4d0f3a78b9fff21cf011454425ff14b9219e1e WatchSource:0}: Error finding container 7bb8905c7f552cd9abe715354c4d0f3a78b9fff21cf011454425ff14b9219e1e: Status 404 returned error can't find the container with id 7bb8905c7f552cd9abe715354c4d0f3a78b9fff21cf011454425ff14b9219e1e Oct 30 09:20:07.230752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.230716 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:07.230879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.230802 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:07.230879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.230849 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:07.231860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.231833 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:20:07.287808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.287777 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 09:20:07.288300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.287827 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 09:20:07.837741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.837516 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" event={"ID":"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19","Type":"ContainerStarted","Data":"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6"} Oct 30 09:20:07.837741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.837591 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" event={"ID":"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19","Type":"ContainerStarted","Data":"7bb8905c7f552cd9abe715354c4d0f3a78b9fff21cf011454425ff14b9219e1e"} Oct 30 09:20:07.938239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:07.937108 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podStartSLOduration=1.937088079 podStartE2EDuration="1.937088079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:07.900977995 +0000 UTC m=+1571.624405185" watchObservedRunningTime="2026-10-30 09:20:07.937088079 +0000 UTC m=+1571.660515223" Oct 30 09:20:08.285299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.283086 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 09:20:08.285299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.283154 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 09:20:08.638333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.634369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:08.760422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.757078 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:08.761721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.761694 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:08.763239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.763199 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:08.763442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.763417 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:08.972849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:08.972758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:09.078643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:09.078607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:09.284918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:09.284811 4920 patch_prober.go:29] interesting pod/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 09:20:09.285792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:09.285176 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 09:20:09.762297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:09.761750 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:09.762297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:09.761807 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:10.290538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.290505 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:20:10.291036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.291000 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:20:10.291887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.291865 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:20:10.762307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.762203 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:10.762307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.762278 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:10.892325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:10.892283 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:11.241189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:11.241157 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:11.339751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:11.339721 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:11.761675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:11.761635 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:11.769461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:11.769313 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:11.795425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:11.795385 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:12.480464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:12.480428 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:12.692071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:12.692033 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:12.761957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:12.761877 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:12.761957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:12.761933 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:13.062524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:13.062489 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:13.317384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:13.317275 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:13.761281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:13.761192 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:13.761436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:13.761275 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:14.118363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:14.118281 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:14.463267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:14.463143 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:14.761717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:14.761625 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 09:20:14.761717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:14.761696 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 09:20:14.799143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:14.799106 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:15.465044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:15.464991 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:20:16.605181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.605142 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:16.762512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.762463 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-8k2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:20:16.762721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.762544 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:20:16.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.826989 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:16.835243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.828590 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:16.842621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:16.842590 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:20:17.547765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:17.547726 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:17.662001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:17.661966 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:18.466171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:18.466137 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:19.284858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:19.284823 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:19.394272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:19.394241 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:20.362813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:20.362783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:21.366120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.366076 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:20:21.382119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.382083 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:20:21.382328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.382200 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480134 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480315 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480357 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480401 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480434 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480465 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480561 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766kk\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480716 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.480793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.480758 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585169 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585241 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585316 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585426 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585458 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585515 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585545 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585573 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.585782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.585682 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766kk\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.607317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.586569 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.619316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.619238 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766kk\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.620177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.620154 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.620840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.620819 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.621181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.621161 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.624346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.621637 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.647318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.641440 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.647950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.647867 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.662951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.662914 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5cq5q\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:21.780532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:21.780502 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:22.032295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:22.032099 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:22.282887 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:20:22.282851 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61562d3d_8726_4942_a7a6_37190abd9f11.slice/crio-64a06ad72554e35273d8905d5b21f53854b2304d944d0f28947ceafa9591a8b6 WatchSource:0}: Error finding container 64a06ad72554e35273d8905d5b21f53854b2304d944d0f28947ceafa9591a8b6: Status 404 returned error can't find the container with id 64a06ad72554e35273d8905d5b21f53854b2304d944d0f28947ceafa9591a8b6 Oct 30 09:20:22.285818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:22.285370 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.285818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:22.285453 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.285818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:22.285482 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.286801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:22.286767 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:20:23.301956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.301544 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" event={"ID":"61562d3d-8726-4942-a7a6-37190abd9f11","Type":"ContainerStarted","Data":"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54"} Oct 30 09:20:23.302451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.301987 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" event={"ID":"61562d3d-8726-4942-a7a6-37190abd9f11","Type":"ContainerStarted","Data":"64a06ad72554e35273d8905d5b21f53854b2304d944d0f28947ceafa9591a8b6"} Oct 30 09:20:23.357042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.356980 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podStartSLOduration=2.3569579640000002 podStartE2EDuration="2.356957964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:23.354832436 +0000 UTC m=+1587.078259580" watchObservedRunningTime="2026-10-30 09:20:23.356957964 +0000 UTC m=+1587.080385109" Oct 30 09:20:23.692754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.692671 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:23.781917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.781886 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:23.783035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.783006 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:23.783161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:23.783066 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:24.781551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:24.781516 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:24.782094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:24.781570 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:25.356349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:25.356272 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:25.452701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:25.452602 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:25.781367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:25.781287 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:25.781546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:25.781402 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:26.782813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:26.782779 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:26.783446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:26.783410 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:27.376077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:27.376048 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:27.781954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:27.781874 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:27.782115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:27.781946 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:28.781421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:28.781336 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:28.781421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:28.781391 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:29.150743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.150693 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:29.476378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.476280 4920 generic.go:356] "Generic (PLEG): container finished" podID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerID="c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4" exitCode=0 Oct 30 09:20:29.476378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.476361 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerDied","Data":"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4"} Oct 30 09:20:29.476901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.476879 4920 scope.go:122] "RemoveContainer" containerID="c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4" Oct 30 09:20:29.478285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478265 4920 generic.go:356] "Generic (PLEG): container finished" podID="58bee755-9e50-4cc3-b825-974fd9cea132" containerID="846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33" exitCode=0 Oct 30 09:20:29.478400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478285 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerDied","Data":"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33"} Oct 30 09:20:29.478608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478582 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.478689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478620 4920 scope.go:122] "RemoveContainer" containerID="846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33" Oct 30 09:20:29.478689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478650 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.478689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.478680 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.480035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.480007 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.480133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.480070 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.480133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.480100 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:29.782233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.782134 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:29.782735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:29.782195 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:30.489031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.488987 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerStarted","Data":"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506"} Oct 30 09:20:30.492233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.492192 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerStarted","Data":"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8"} Oct 30 09:20:30.594128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.594053 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:20:30.758338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.757008 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:30.781270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.781231 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 09:20:30.781441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.781313 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 09:20:30.865240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.858611 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:20:30.865240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.858754 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:30.865240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.858802 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:30.915237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.907677 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:20:30.915237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.907815 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:30.915237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:30.907873 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:31.191056 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:20:31.191018 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:20:31.872245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:31.870671 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:31.872245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:31.870754 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:31.908256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:31.905661 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:31.908256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:31.905721 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:32.259831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.259738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:32.532663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.532576 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:32.581026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.580971 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:32.587674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.587589 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:20:32.871272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.869634 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:32.871272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.869699 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:32.936243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.929365 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:32.936243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:32.929432 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:33.811303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.810567 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:33.864244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.863628 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:33.864244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.863685 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:33.929241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.928420 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:33.953244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.929498 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:33.953244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:33.929550 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:34.033241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:34.025776 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:34.861249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:34.860620 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:34.861249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:34.860682 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:34.906242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:34.905771 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:34.906242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:34.905827 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:35.469183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:35.469146 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:35.857028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:35.856993 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 09:20:35.857211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:35.857050 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 09:20:35.905592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:35.905543 4920 patch_prober.go:29] interesting pod/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 09:20:35.905792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:35.905628 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 09:20:36.187895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:36.187812 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:20:36.861623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:36.861585 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:20:36.862585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:36.862562 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:20:36.909650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:36.909622 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:20:36.910721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:36.910690 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:20:37.143960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:37.143869 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:38.659720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:38.659636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:40.171628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:40.171595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:41.660545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:41.660506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:43.141775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:43.141734 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:44.760181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:44.760147 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:46.562889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:46.562837 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:48.674455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:48.674420 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:49.088072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:49.088036 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:49.285753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:49.285718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:49.400269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:49.400170 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:50.632504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:50.632464 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:51.080673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.080626 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72"] Oct 30 09:20:51.085397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.085376 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.105981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.105940 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72"] Oct 30 09:20:51.108742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.108716 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:20:51.108742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.108716 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:20:51.108923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.108815 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:20:51.108923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.108822 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:20:51.229407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.229377 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.229589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.229526 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v98d\" (UniqueName: \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.229589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.229579 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.331021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.330935 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.331184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.331044 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v98d\" (UniqueName: \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.331184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.331087 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.331773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.331750 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.333482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.333459 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.357305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.357274 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v98d\" (UniqueName: \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\") pod \"maas-api-revoke-keys-e2e-api-a-648892-q7t72\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.402319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.402282 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:51.707949 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:20:51.707917 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d83716d_a13a_4b2f_b80c_b2ab959fcd32.slice/crio-d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5 WatchSource:0}: Error finding container d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5: Status 404 returned error can't find the container with id d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5 Oct 30 09:20:51.711571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.711547 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72"] Oct 30 09:20:51.930299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:51.930263 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" event={"ID":"7d83716d-a13a-4b2f-b80c-b2ab959fcd32","Type":"ContainerStarted","Data":"d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5"} Oct 30 09:20:52.590832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:52.590784 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:52.942272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:52.942139 4920 generic.go:356] "Generic (PLEG): container finished" podID="7d83716d-a13a-4b2f-b80c-b2ab959fcd32" containerID="b51aea464be017661b1344896bad835bd3eacfc52b6be3a6e99339900216d742" exitCode=0 Oct 30 09:20:52.942731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:52.942274 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" event={"ID":"7d83716d-a13a-4b2f-b80c-b2ab959fcd32","Type":"ContainerDied","Data":"b51aea464be017661b1344896bad835bd3eacfc52b6be3a6e99339900216d742"} Oct 30 09:20:53.038849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:53.038811 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6"] Oct 30 09:20:53.043975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:53.043946 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a1d587-4hwp6"] Oct 30 09:20:54.207616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.207510 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:54.290278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.289235 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\" (UniqueName: \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\") pod \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " Oct 30 09:20:54.290278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.289421 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\") pod \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " Oct 30 09:20:54.290278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.289462 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\") pod \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\" (UID: \"7d83716d-a13a-4b2f-b80c-b2ab959fcd32\") " Oct 30 09:20:54.290770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.290747 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca" pod "7d83716d-a13a-4b2f-b80c-b2ab959fcd32" (UID: "7d83716d-a13a-4b2f-b80c-b2ab959fcd32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:20:54.317882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.317847 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d" pod "7d83716d-a13a-4b2f-b80c-b2ab959fcd32" (UID: "7d83716d-a13a-4b2f-b80c-b2ab959fcd32"). InnerVolumeSpecName "kube-api-access-4v98d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:20:54.343248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.337483 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca" pod "7d83716d-a13a-4b2f-b80c-b2ab959fcd32" (UID: "7d83716d-a13a-4b2f-b80c-b2ab959fcd32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:20:54.391314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.391275 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v98d\" (UniqueName: \"kubernetes.io/projected/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-kube-api-access-4v98d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:54.391314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.391313 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:54.391543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.391328 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d83716d-a13a-4b2f-b80c-b2ab959fcd32-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:20:54.400979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.400938 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2" path="/var/lib/kubelet/pods/7a70b0e9-a169-4ee5-9cd0-4872a7c90fc2/volumes" Oct 30 09:20:54.510083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.509996 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:54.962926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.962896 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" Oct 30 09:20:54.963114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.962945 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72" event={"ID":"7d83716d-a13a-4b2f-b80c-b2ab959fcd32","Type":"ContainerDied","Data":"d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5"} Oct 30 09:20:54.963114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:54.962990 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d241ee4f4665397b2a429ccbdae847313179f935c31d7fddf273724f864fbfa5" Oct 30 09:20:55.029606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:55.029567 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2"] Oct 30 09:20:55.035612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:55.035574 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cc9eeb-xvmm2"] Oct 30 09:20:56.127438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:56.127398 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:56.400153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:56.400064 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a518c097-78d5-4887-b2e2-b10a34bd4e7a" path="/var/lib/kubelet/pods/a518c097-78d5-4887-b2e2-b10a34bd4e7a/volumes" Oct 30 09:20:57.591568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:57.591538 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:58.331701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:58.331638 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:59.188489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.188448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:20:59.222811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.222767 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn"] Oct 30 09:20:59.223493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.223474 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7d83716d-a13a-4b2f-b80c-b2ab959fcd32" containerName="revoke-keys" Oct 30 09:20:59.223493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.223491 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d83716d-a13a-4b2f-b80c-b2ab959fcd32" containerName="revoke-keys" Oct 30 09:20:59.223662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.223626 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d83716d-a13a-4b2f-b80c-b2ab959fcd32" containerName="revoke-keys" Oct 30 09:20:59.228827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.228806 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.259508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.259459 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:20:59.260147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.260115 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:20:59.260337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.260294 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:20:59.260414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.260394 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:20:59.289297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.280872 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn"] Oct 30 09:20:59.295128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.295091 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7cv\" (UniqueName: \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.295462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.295416 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.295748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.295710 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.397121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.397083 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7cv\" (UniqueName: \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.397121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.397127 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.397412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.397270 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.398018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.397995 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.399648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.399613 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.418657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.418625 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7cv\" (UniqueName: \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\") pod \"maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.584060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.584029 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:20:59.897827 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:20:59.897796 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806bc361_67f2_4d54_859c_af9354dd1b2a.slice/crio-80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f WatchSource:0}: Error finding container 80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f: Status 404 returned error can't find the container with id 80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f Oct 30 09:20:59.902087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:20:59.902057 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn"] Oct 30 09:21:00.011865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.011827 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" event={"ID":"806bc361-67f2-4d54-859c-af9354dd1b2a","Type":"ContainerStarted","Data":"80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f"} Oct 30 09:21:00.453603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.453567 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2"] Oct 30 09:21:00.458148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.458093 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.475726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.475698 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2"] Oct 30 09:21:00.513241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.513180 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.513470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.513444 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.513685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.513664 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzzx\" (UniqueName: \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.615075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.615046 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.615271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.615175 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.615339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.615298 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzzx\" (UniqueName: \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.615816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.615790 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.617342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.617319 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.644686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.644649 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzzx\" (UniqueName: \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\") pod \"maas-api-revoke-sub-e035ff14-2mqf2\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.774069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.773993 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:00.866356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:00.866321 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:01.026944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.026838 4920 generic.go:356] "Generic (PLEG): container finished" podID="806bc361-67f2-4d54-859c-af9354dd1b2a" containerID="d39be6eea88caa90ff39febdc591787ea8ef484cdd9c4a58ee1a9456d0c50e6c" exitCode=0 Oct 30 09:21:01.027313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.027275 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" event={"ID":"806bc361-67f2-4d54-859c-af9354dd1b2a","Type":"ContainerDied","Data":"d39be6eea88caa90ff39febdc591787ea8ef484cdd9c4a58ee1a9456d0c50e6c"} Oct 30 09:21:01.034729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.034690 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2"] Oct 30 09:21:01.041290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.041259 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff1106c7-554j2"] Oct 30 09:21:01.084453 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:21:01.084414 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9c0e23_6b99_4f81_a475_85a90dda3157.slice/crio-ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115 WatchSource:0}: Error finding container ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115: Status 404 returned error can't find the container with id ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115 Oct 30 09:21:01.095242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.095203 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2"] Oct 30 09:21:01.150990 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:01.150950 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:21:01.745265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.745133 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:01.858736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:01.858658 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:02.031373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.031292 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj"] Oct 30 09:21:02.037819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.037786 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk"] Oct 30 09:21:02.041474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.041402 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" event={"ID":"3a9c0e23-6b99-4f81-a475-85a90dda3157","Type":"ContainerStarted","Data":"b47e6bd5ead5c68d822f9dedd37ce93c42a738587f14308322534bd91315f25b"} Oct 30 09:21:02.041474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.041460 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" event={"ID":"3a9c0e23-6b99-4f81-a475-85a90dda3157","Type":"ContainerStarted","Data":"ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115"} Oct 30 09:21:02.044901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.044876 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-645924b4-bshpj"] Oct 30 09:21:02.051115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.051088 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987b2411-8b2tk"] Oct 30 09:21:02.084172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.084116 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" podStartSLOduration=2.084098965 podStartE2EDuration="2.084098965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:02.077632599 +0000 UTC m=+1625.801059742" watchObservedRunningTime="2026-10-30 09:21:02.084098965 +0000 UTC m=+1625.807526120" Oct 30 09:21:02.226920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.226893 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:21:02.379787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.379747 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\") pod \"806bc361-67f2-4d54-859c-af9354dd1b2a\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " Oct 30 09:21:02.380102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.380080 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\" (UniqueName: \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\") pod \"806bc361-67f2-4d54-859c-af9354dd1b2a\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " Oct 30 09:21:02.384239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.380248 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\") pod \"806bc361-67f2-4d54-859c-af9354dd1b2a\" (UID: \"806bc361-67f2-4d54-859c-af9354dd1b2a\") " Oct 30 09:21:02.402322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.388203 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca" pod "806bc361-67f2-4d54-859c-af9354dd1b2a" (UID: "806bc361-67f2-4d54-859c-af9354dd1b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:02.488310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.488265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:02.490685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.490565 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:02.534381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.527109 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv" pod "806bc361-67f2-4d54-859c-af9354dd1b2a" (UID: "806bc361-67f2-4d54-859c-af9354dd1b2a"). InnerVolumeSpecName "kube-api-access-cf7cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:02.534381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.529720 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca" pod "806bc361-67f2-4d54-859c-af9354dd1b2a" (UID: "806bc361-67f2-4d54-859c-af9354dd1b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:02.549002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.543606 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fcba3c-543f-4ad9-99fb-43fd7ef88057" path="/var/lib/kubelet/pods/90fcba3c-543f-4ad9-99fb-43fd7ef88057/volumes" Oct 30 09:21:02.549002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.544201 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda5aa14-b10a-48c3-8b2b-5e61752471e1" path="/var/lib/kubelet/pods/cda5aa14-b10a-48c3-8b2b-5e61752471e1/volumes" Oct 30 09:21:02.549002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.548627 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd93536-8df5-4d75-ba29-0b46b169d5a1" path="/var/lib/kubelet/pods/edd93536-8df5-4d75-ba29-0b46b169d5a1/volumes" Oct 30 09:21:02.613303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.613274 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806bc361-67f2-4d54-859c-af9354dd1b2a-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:02.613303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:02.613298 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7cv\" (UniqueName: \"kubernetes.io/projected/806bc361-67f2-4d54-859c-af9354dd1b2a-kube-api-access-cf7cv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:03.051617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:03.051580 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" Oct 30 09:21:03.052053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:03.051612 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn" event={"ID":"806bc361-67f2-4d54-859c-af9354dd1b2a","Type":"ContainerDied","Data":"80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f"} Oct 30 09:21:03.052053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:03.051673 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bc3615ab265efff57162e487a3ec7e2cf043a87959c835b5819b3763f9608f" Oct 30 09:21:03.053824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:03.053796 4920 generic.go:356] "Generic (PLEG): container finished" podID="3a9c0e23-6b99-4f81-a475-85a90dda3157" containerID="b47e6bd5ead5c68d822f9dedd37ce93c42a738587f14308322534bd91315f25b" exitCode=0 Oct 30 09:21:03.053984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:03.053839 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" event={"ID":"3a9c0e23-6b99-4f81-a475-85a90dda3157","Type":"ContainerDied","Data":"b47e6bd5ead5c68d822f9dedd37ce93c42a738587f14308322534bd91315f25b"} Oct 30 09:21:04.231678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.231650 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:04.240676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.240648 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:04.341626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.339434 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\" (UniqueName: \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\") pod \"3a9c0e23-6b99-4f81-a475-85a90dda3157\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " Oct 30 09:21:04.341626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.339483 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\") pod \"3a9c0e23-6b99-4f81-a475-85a90dda3157\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " Oct 30 09:21:04.341626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.339570 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\") pod \"3a9c0e23-6b99-4f81-a475-85a90dda3157\" (UID: \"3a9c0e23-6b99-4f81-a475-85a90dda3157\") " Oct 30 09:21:04.341626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.341394 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca" pod "3a9c0e23-6b99-4f81-a475-85a90dda3157" (UID: "3a9c0e23-6b99-4f81-a475-85a90dda3157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:04.343576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.343543 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca" pod "3a9c0e23-6b99-4f81-a475-85a90dda3157" (UID: "3a9c0e23-6b99-4f81-a475-85a90dda3157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:04.343741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.343697 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx" pod "3a9c0e23-6b99-4f81-a475-85a90dda3157" (UID: "3a9c0e23-6b99-4f81-a475-85a90dda3157"). InnerVolumeSpecName "kube-api-access-xxzzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:04.442011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.441972 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:04.442011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.442009 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzzx\" (UniqueName: \"kubernetes.io/projected/3a9c0e23-6b99-4f81-a475-85a90dda3157-kube-api-access-xxzzx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:04.442011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:04.442019 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9c0e23-6b99-4f81-a475-85a90dda3157-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:05.072842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:05.072809 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" Oct 30 09:21:05.073027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:05.072836 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2" event={"ID":"3a9c0e23-6b99-4f81-a475-85a90dda3157","Type":"ContainerDied","Data":"ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115"} Oct 30 09:21:05.073027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:05.072907 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2e110624184012298d1869d68f9bfbad330fc5920321135bd09e1fa85d4115" Oct 30 09:21:05.775834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:05.775797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:07.357923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:07.357893 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:09.080514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:09.080476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:09.815444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:09.815406 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:09.915307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:09.915273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:10.863687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:10.863646 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:10.961602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:10.961561 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:12.453696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:12.453667 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:13.997699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:13.997661 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:15.031850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:15.031815 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp"] Oct 30 09:21:15.044930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:15.044897 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48dc1c8-j72tp"] Oct 30 09:21:15.596899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:15.596868 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:15.914473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:15.914383 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:21:16.402514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:16.402476 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d700e465-3d43-40c2-baf4-f740a4cd37b2" path="/var/lib/kubelet/pods/d700e465-3d43-40c2-baf4-f740a4cd37b2/volumes" Oct 30 09:21:17.217657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:17.217611 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:17.319470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:17.319435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:18.873321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:18.873287 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:19.033140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:19.033103 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2"] Oct 30 09:21:19.043981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:19.043950 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b148c14b-s2sb2"] Oct 30 09:21:20.399740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:20.399698 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1947729-466b-486c-a88f-65c71ef39fbb" path="/var/lib/kubelet/pods/d1947729-466b-486c-a88f-65c71ef39fbb/volumes" Oct 30 09:21:20.470162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:20.470121 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:22.063940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:22.063910 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:23.616964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:23.616927 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:25.148101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:25.148054 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:26.533793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:26.533752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:27.211482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.211424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:27.311912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.311878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:27.343664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.343628 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72"] Oct 30 09:21:27.355401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.355371 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-648892-q7t72"] Oct 30 09:21:27.415514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.415304 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:27.847574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:27.847539 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:28.399051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:28.399016 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d83716d-a13a-4b2f-b80c-b2ab959fcd32" path="/var/lib/kubelet/pods/7d83716d-a13a-4b2f-b80c-b2ab959fcd32/volumes" Oct 30 09:21:29.068562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:29.068525 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:30.033753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:30.033717 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr"] Oct 30 09:21:30.048593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:30.048562 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2c8dba-tbcvr"] Oct 30 09:21:30.399061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:30.399027 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed18c35b-2c79-4a9f-b0e8-ede7198ce194" path="/var/lib/kubelet/pods/ed18c35b-2c79-4a9f-b0e8-ede7198ce194/volumes" Oct 30 09:21:30.654699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:30.654596 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:31.140102 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:31.140067 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:21:32.342832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:32.342799 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:33.878424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.878386 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl"] Oct 30 09:21:33.879059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879044 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3a9c0e23-6b99-4f81-a475-85a90dda3157" containerName="revoke-keys" Oct 30 09:21:33.879105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879060 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9c0e23-6b99-4f81-a475-85a90dda3157" containerName="revoke-keys" Oct 30 09:21:33.879105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879085 4920 cpu_manager.go:414] "containerMap: removing container" podUID="806bc361-67f2-4d54-859c-af9354dd1b2a" containerName="revoke-keys" Oct 30 09:21:33.879105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879090 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806bc361-67f2-4d54-859c-af9354dd1b2a" containerName="revoke-keys" Oct 30 09:21:33.879204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879194 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9c0e23-6b99-4f81-a475-85a90dda3157" containerName="revoke-keys" Oct 30 09:21:33.879254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.879208 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="806bc361-67f2-4d54-859c-af9354dd1b2a" containerName="revoke-keys" Oct 30 09:21:33.884611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.884588 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:33.885368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.885342 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:33.901016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.900975 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl"] Oct 30 09:21:33.918613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.918575 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:21:33.918825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.918723 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:21:33.918825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.918779 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:21:33.918948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.918928 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:21:33.991448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.986856 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vckd\" (UniqueName: \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:33.991448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.986968 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:33.991448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:33.987131 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.087948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.087913 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:34.088370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.088345 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vckd\" (UniqueName: \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.088491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.088437 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.088595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.088579 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.089098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.089080 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.090830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.090813 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.109115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.109086 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vckd\" (UniqueName: \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.192907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.192822 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:34.200542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.200510 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:34.406474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.405153 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:21:34.406474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.405422 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" containerID="cri-o://7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506" gracePeriod=30 Oct 30 09:21:34.417608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.417580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:34.573784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.573748 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl"] Oct 30 09:21:34.973332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.973297 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p"] Oct 30 09:21:34.979350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.979328 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:34.995759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:34.995732 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p"] Oct 30 09:21:35.034013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.033973 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.034254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.034035 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.034254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.034145 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5zc\" (UniqueName: \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.135585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.135549 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5zc\" (UniqueName: \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.135798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.135782 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.135848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.135836 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.136641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.136609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.138298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.138270 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.154099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.154076 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5zc\" (UniqueName: \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\") pod \"maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.296055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.296025 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:35.388707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.387869 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" event={"ID":"8042310a-ae8e-4981-b067-4f646f8dbbed","Type":"ContainerStarted","Data":"c45cce240c5095af8963f0ea24ccafe53db453c10df6924516b384178f3aba27"} Oct 30 09:21:35.388707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.387974 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" event={"ID":"8042310a-ae8e-4981-b067-4f646f8dbbed","Type":"ContainerStarted","Data":"2b8300aabdc4df7df3d600d179fdf71c33f4c364e9f00f027c9445c0d4b83678"} Oct 30 09:21:35.427922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.427855 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" podStartSLOduration=2.427834279 podStartE2EDuration="2.427834279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:35.427431785 +0000 UTC m=+1659.150858929" watchObservedRunningTime="2026-10-30 09:21:35.427834279 +0000 UTC m=+1659.151261424" Oct 30 09:21:35.513308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.513270 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:35.608881 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:21:35.608851 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172ab02a_26c0_44e3_b8e3_aecdd01b21f3.slice/crio-3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744 WatchSource:0}: Error finding container 3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744: Status 404 returned error can't find the container with id 3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744 Oct 30 09:21:35.611828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:35.611805 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p"] Oct 30 09:21:36.237893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.237854 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:36.379537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.379460 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:36.400818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.400784 4920 generic.go:356] "Generic (PLEG): container finished" podID="8042310a-ae8e-4981-b067-4f646f8dbbed" containerID="c45cce240c5095af8963f0ea24ccafe53db453c10df6924516b384178f3aba27" exitCode=0 Oct 30 09:21:36.403966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.403929 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" event={"ID":"8042310a-ae8e-4981-b067-4f646f8dbbed","Type":"ContainerDied","Data":"c45cce240c5095af8963f0ea24ccafe53db453c10df6924516b384178f3aba27"} Oct 30 09:21:36.404141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.403973 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" event={"ID":"172ab02a-26c0-44e3-b8e3-aecdd01b21f3","Type":"ContainerStarted","Data":"e45e11408b91265e3f91c8bb93be68b70665d13a774b55d1d773268138e1b1bb"} Oct 30 09:21:36.404141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.403998 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" event={"ID":"172ab02a-26c0-44e3-b8e3-aecdd01b21f3","Type":"ContainerStarted","Data":"3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744"} Oct 30 09:21:36.437782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.437745 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn"] Oct 30 09:21:36.454025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.453994 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca0df2-nwvpn"] Oct 30 09:21:36.480502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.480449 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" podStartSLOduration=2.480434378 podStartE2EDuration="2.480434378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:36.478726609 +0000 UTC m=+1660.202153747" watchObservedRunningTime="2026-10-30 09:21:36.480434378 +0000 UTC m=+1660.203861521" Oct 30 09:21:36.482965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.482935 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:36.864331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.864284 4920 patch_prober.go:29] interesting pod/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:21:36.864513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.864350 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:21:36.917610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:36.917578 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:37.415319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.415281 4920 generic.go:356] "Generic (PLEG): container finished" podID="172ab02a-26c0-44e3-b8e3-aecdd01b21f3" containerID="e45e11408b91265e3f91c8bb93be68b70665d13a774b55d1d773268138e1b1bb" exitCode=0 Oct 30 09:21:37.415722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.415326 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" event={"ID":"172ab02a-26c0-44e3-b8e3-aecdd01b21f3","Type":"ContainerDied","Data":"e45e11408b91265e3f91c8bb93be68b70665d13a774b55d1d773268138e1b1bb"} Oct 30 09:21:37.797498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.797465 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:37.884480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.884447 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\") pod \"8042310a-ae8e-4981-b067-4f646f8dbbed\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " Oct 30 09:21:37.884677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.884619 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\") pod \"8042310a-ae8e-4981-b067-4f646f8dbbed\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " Oct 30 09:21:37.884751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.884720 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\" (UniqueName: \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\") pod \"8042310a-ae8e-4981-b067-4f646f8dbbed\" (UID: \"8042310a-ae8e-4981-b067-4f646f8dbbed\") " Oct 30 09:21:37.885165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.885136 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca" pod "8042310a-ae8e-4981-b067-4f646f8dbbed" (UID: "8042310a-ae8e-4981-b067-4f646f8dbbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:37.885446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.885427 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:37.886798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.886771 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca" pod "8042310a-ae8e-4981-b067-4f646f8dbbed" (UID: "8042310a-ae8e-4981-b067-4f646f8dbbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:37.886975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.886949 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd" pod "8042310a-ae8e-4981-b067-4f646f8dbbed" (UID: "8042310a-ae8e-4981-b067-4f646f8dbbed"). InnerVolumeSpecName "kube-api-access-7vckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:37.987271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.987201 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vckd\" (UniqueName: \"kubernetes.io/projected/8042310a-ae8e-4981-b067-4f646f8dbbed-kube-api-access-7vckd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:37.987271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:37.987263 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042310a-ae8e-4981-b067-4f646f8dbbed-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:38.409346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.403625 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806bc361-67f2-4d54-859c-af9354dd1b2a" path="/var/lib/kubelet/pods/806bc361-67f2-4d54-859c-af9354dd1b2a/volumes" Oct 30 09:21:38.428449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.428415 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" Oct 30 09:21:38.431954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.429393 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl" event={"ID":"8042310a-ae8e-4981-b067-4f646f8dbbed","Type":"ContainerDied","Data":"2b8300aabdc4df7df3d600d179fdf71c33f4c364e9f00f027c9445c0d4b83678"} Oct 30 09:21:38.431954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.429448 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b8300aabdc4df7df3d600d179fdf71c33f4c364e9f00f027c9445c0d4b83678" Oct 30 09:21:38.621440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.621293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:38.683555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.683527 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:38.847758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.845570 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\") pod \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " Oct 30 09:21:38.847758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.845888 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\" (UniqueName: \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\") pod \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " Oct 30 09:21:38.847758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.846029 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\") pod \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\" (UID: \"172ab02a-26c0-44e3-b8e3-aecdd01b21f3\") " Oct 30 09:21:38.849665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.848954 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca" pod "172ab02a-26c0-44e3-b8e3-aecdd01b21f3" (UID: "172ab02a-26c0-44e3-b8e3-aecdd01b21f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:38.863475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.863442 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:38.890367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.890320 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca" pod "172ab02a-26c0-44e3-b8e3-aecdd01b21f3" (UID: "172ab02a-26c0-44e3-b8e3-aecdd01b21f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:38.896029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.895788 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc" pod "172ab02a-26c0-44e3-b8e3-aecdd01b21f3" (UID: "172ab02a-26c0-44e3-b8e3-aecdd01b21f3"). InnerVolumeSpecName "kube-api-access-vz5zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:38.965895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.965805 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5zc\" (UniqueName: \"kubernetes.io/projected/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-kube-api-access-vz5zc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:38.965895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:38.965846 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ab02a-26c0-44e3-b8e3-aecdd01b21f3-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:39.466064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:39.466033 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" Oct 30 09:21:39.466549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:39.466035 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p" event={"ID":"172ab02a-26c0-44e3-b8e3-aecdd01b21f3","Type":"ContainerDied","Data":"3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744"} Oct 30 09:21:39.466549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:39.466180 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bf4628da36577c843f3ae6d30224c0138edc5057364882d83cfbcea0e299744" Oct 30 09:21:40.276688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.276655 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:21:40.288650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.288626 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.288804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.288665 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.288804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.288732 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.288955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.288890 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:40.289019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.288944 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.289074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289052 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.289129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289099 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.289197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289177 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:40.289290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289181 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.289411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289386 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.289497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289448 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:40.289543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289494 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:40.289612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289594 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:40.289667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.289655 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\") pod \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\" (UID: \"eaff7f8b-9859-4e8c-8fd0-293c44098d95\") " Oct 30 09:21:40.290323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.290301 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.290584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.290469 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.290748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.290589 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.290748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.290607 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.290748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.290622 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.291266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.291240 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:40.291363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.291274 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:40.291686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.291665 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:21:40.291836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.291819 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq" pod "eaff7f8b-9859-4e8c-8fd0-293c44098d95" (UID: "eaff7f8b-9859-4e8c-8fd0-293c44098d95"). InnerVolumeSpecName "kube-api-access-lpsdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:40.382272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.382173 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:40.392775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.392743 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.392775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.392778 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.392992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.392793 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaff7f8b-9859-4e8c-8fd0-293c44098d95-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.392992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.392808 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsdq\" (UniqueName: \"kubernetes.io/projected/eaff7f8b-9859-4e8c-8fd0-293c44098d95-kube-api-access-lpsdq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:40.510537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.509124 4920 generic.go:356] "Generic (PLEG): container finished" podID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerID="7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506" exitCode=0 Oct 30 09:21:40.510537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.509208 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerDied","Data":"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506"} Oct 30 09:21:40.510537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.509276 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" event={"ID":"eaff7f8b-9859-4e8c-8fd0-293c44098d95","Type":"ContainerDied","Data":"48baf9a61f1f971d6210581a6dbb917a97cde4894370588b78534473aa8f52a9"} Oct 30 09:21:40.510537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.509310 4920 scope.go:122] "RemoveContainer" containerID="7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506" Oct 30 09:21:40.510537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.509489 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn" Oct 30 09:21:40.546779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.546396 4920 scope.go:122] "RemoveContainer" containerID="c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4" Oct 30 09:21:40.590993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.589177 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:21:40.624411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.620412 4920 scope.go:122] "RemoveContainer" containerID="7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506" Oct 30 09:21:40.630947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.625711 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-648892-openshift-default-7c4b976b7-hr9wn"] Oct 30 09:21:40.664331 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:40.632840 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506\": container with ID starting with 7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506 not found: ID does not exist" containerID="7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506" Oct 30 09:21:40.664331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.632884 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506"} err="failed to get container status \"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506\": rpc error: code = NotFound desc = could not find container \"7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506\": container with ID starting with 7156860511ef46150a5e906cf1f1cccd0a74d841f93eb0c9c1ab3e43ca34b506 not found: ID does not exist" Oct 30 09:21:40.664331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.632913 4920 scope.go:122] "RemoveContainer" containerID="c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4" Oct 30 09:21:40.664331 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:40.633424 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4\": container with ID starting with c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4 not found: ID does not exist" containerID="c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4" Oct 30 09:21:40.664331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:40.633463 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4"} err="failed to get container status \"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4\": rpc error: code = NotFound desc = could not find container \"c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4\": container with ID starting with c14dfedb1abe63cac4f21956c5847469ba258438f9cbdcd3650c98a2863124b4 not found: ID does not exist" Oct 30 09:21:42.145741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:42.145704 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:42.401411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:42.401315 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" path="/var/lib/kubelet/pods/eaff7f8b-9859-4e8c-8fd0-293c44098d95/volumes" Oct 30 09:21:42.831709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:42.831665 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:42.948006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:42.947834 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:43.181005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.180913 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:21:43.181537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.181192 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" containerID="cri-o://650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8" gracePeriod=30 Oct 30 09:21:43.203596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.203564 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:43.791255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791205 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v"] Oct 30 09:21:43.791898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791884 4920 cpu_manager.go:414] "containerMap: removing container" podUID="172ab02a-26c0-44e3-b8e3-aecdd01b21f3" containerName="revoke-keys" Oct 30 09:21:43.791960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791899 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172ab02a-26c0-44e3-b8e3-aecdd01b21f3" containerName="revoke-keys" Oct 30 09:21:43.791960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791930 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8042310a-ae8e-4981-b067-4f646f8dbbed" containerName="revoke-keys" Oct 30 09:21:43.791960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791935 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8042310a-ae8e-4981-b067-4f646f8dbbed" containerName="revoke-keys" Oct 30 09:21:43.791960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791953 4920 cpu_manager.go:414] "containerMap: removing container" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:43.791960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791958 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:43.792118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791976 4920 cpu_manager.go:414] "containerMap: removing container" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:43.792118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.791981 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:43.792118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.792068 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:43.792118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.792078 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="172ab02a-26c0-44e3-b8e3-aecdd01b21f3" containerName="revoke-keys" Oct 30 09:21:43.792118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.792087 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8042310a-ae8e-4981-b067-4f646f8dbbed" containerName="revoke-keys" Oct 30 09:21:43.797406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.797385 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:43.815772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.815741 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v"] Oct 30 09:21:43.817442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.817416 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:21:43.817582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.817481 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:21:43.817582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.817569 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:21:43.823676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.823646 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:21:43.911345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.911310 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:43.911549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.911426 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:43.911549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.911516 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tfl\" (UniqueName: \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:43.930463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:43.930432 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:44.023241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.020042 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.023241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.020179 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.023241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.020321 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tfl\" (UniqueName: \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.032919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.028506 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.032919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.030921 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.042231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.041668 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tfl\" (UniqueName: \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\") pod \"maas-api-revoke-keys-e2e-api-b-8d1106-8787v\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.114977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.114939 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:44.275938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.275894 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:44.375055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.374966 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:44.477762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.477725 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v"] Oct 30 09:21:44.553794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.553757 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" event={"ID":"5c5054a9-5ceb-4bc5-b694-588624ecbb67","Type":"ContainerStarted","Data":"c14b155223c47df4df0f51fe12d675c68d3fdfafe3764e311ea8666a67943e33"} Oct 30 09:21:44.691945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.691864 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6"] Oct 30 09:21:44.692673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.692656 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaff7f8b-9859-4e8c-8fd0-293c44098d95" containerName="istio-proxy" Oct 30 09:21:44.697656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.697631 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.722623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.719781 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6"] Oct 30 09:21:44.848238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.847081 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.848238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.847212 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.848238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.847345 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfmd\" (UniqueName: \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.949441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.949348 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.949786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.949761 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.949984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.949967 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfmd\" (UniqueName: \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.951822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.951796 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.978266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.968975 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:44.978266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:44.972941 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfmd\" (UniqueName: \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\") pod \"maas-api-revoke-sub-1c94b2eb-4glg6\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:45.029797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.029755 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:45.349192 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:21:45.349155 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815ccca2_b75f_42fa_967a_0fcb6536d805.slice/crio-3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f WatchSource:0}: Error finding container 3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f: Status 404 returned error can't find the container with id 3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f Oct 30 09:21:45.352651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.352629 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6"] Oct 30 09:21:45.578270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.568753 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" event={"ID":"5c5054a9-5ceb-4bc5-b694-588624ecbb67","Type":"ContainerStarted","Data":"a3826878c11291583bb28a0a240b48d2ce6575ec398e22e67d5024243dfd488a"} Oct 30 09:21:45.578270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.570902 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:45.578270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.571038 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" event={"ID":"815ccca2-b75f-42fa-967a-0fcb6536d805","Type":"ContainerStarted","Data":"3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f"} Oct 30 09:21:45.617531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.617480 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" podStartSLOduration=2.6174625000000002 podStartE2EDuration="2.6174625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:45.605322576 +0000 UTC m=+1669.328749719" watchObservedRunningTime="2026-10-30 09:21:45.6174625 +0000 UTC m=+1669.340889645" Oct 30 09:21:45.688768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.688700 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:45.796083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:45.796046 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:46.581937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:46.581839 4920 generic.go:356] "Generic (PLEG): container finished" podID="815ccca2-b75f-42fa-967a-0fcb6536d805" containerID="a09c96736e89956855cb9c9640a67c201471d00e02b935fc0fb9e831fac6ec83" exitCode=0 Oct 30 09:21:46.581937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:46.581879 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" event={"ID":"815ccca2-b75f-42fa-967a-0fcb6536d805","Type":"ContainerDied","Data":"a09c96736e89956855cb9c9640a67c201471d00e02b935fc0fb9e831fac6ec83"} Oct 30 09:21:46.583866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:46.583837 4920 generic.go:356] "Generic (PLEG): container finished" podID="5c5054a9-5ceb-4bc5-b694-588624ecbb67" containerID="a3826878c11291583bb28a0a240b48d2ce6575ec398e22e67d5024243dfd488a" exitCode=0 Oct 30 09:21:46.583987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:46.583877 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" event={"ID":"5c5054a9-5ceb-4bc5-b694-588624ecbb67","Type":"ContainerDied","Data":"a3826878c11291583bb28a0a240b48d2ce6575ec398e22e67d5024243dfd488a"} Oct 30 09:21:47.050678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.050642 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59"] Oct 30 09:21:47.055237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.055200 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.073573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.073540 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59"] Oct 30 09:21:47.208102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.208056 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.208308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.208204 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cvb\" (UniqueName: \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.208353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.208321 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.310248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.310126 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.310410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.310255 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cvb\" (UniqueName: \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.310410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.310299 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.311140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.311112 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.312782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.312759 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.334565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.334529 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cvb\" (UniqueName: \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\") pod \"maas-api-revoke-sub-23df2ebe-h8f59\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.369781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.369750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:47.372756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.372730 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:47.733808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.733763 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59"] Oct 30 09:21:47.796903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.796874 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:47.832822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.832790 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:47.924807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.924710 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\") pod \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " Oct 30 09:21:47.924977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.924844 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\") pod \"815ccca2-b75f-42fa-967a-0fcb6536d805\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " Oct 30 09:21:47.924977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.924920 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\" (UniqueName: \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\") pod \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " Oct 30 09:21:47.925099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925004 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\" (UniqueName: \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\") pod \"815ccca2-b75f-42fa-967a-0fcb6536d805\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " Oct 30 09:21:47.925099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925079 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca" pod "5c5054a9-5ceb-4bc5-b694-588624ecbb67" (UID: "5c5054a9-5ceb-4bc5-b694-588624ecbb67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:47.925201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925135 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\") pod \"815ccca2-b75f-42fa-967a-0fcb6536d805\" (UID: \"815ccca2-b75f-42fa-967a-0fcb6536d805\") " Oct 30 09:21:47.925299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925206 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\") pod \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\" (UID: \"5c5054a9-5ceb-4bc5-b694-588624ecbb67\") " Oct 30 09:21:47.925572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925550 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca" pod "815ccca2-b75f-42fa-967a-0fcb6536d805" (UID: "815ccca2-b75f-42fa-967a-0fcb6536d805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:47.925928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925870 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:47.926064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.925932 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:47.998056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.998018 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca" pod "815ccca2-b75f-42fa-967a-0fcb6536d805" (UID: "815ccca2-b75f-42fa-967a-0fcb6536d805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:47.998056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.998050 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl" pod "5c5054a9-5ceb-4bc5-b694-588624ecbb67" (UID: "5c5054a9-5ceb-4bc5-b694-588624ecbb67"). InnerVolumeSpecName "kube-api-access-z5tfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:47.998293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.998020 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca" pod "5c5054a9-5ceb-4bc5-b694-588624ecbb67" (UID: "5c5054a9-5ceb-4bc5-b694-588624ecbb67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:47.998293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:47.998094 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd" pod "815ccca2-b75f-42fa-967a-0fcb6536d805" (UID: "815ccca2-b75f-42fa-967a-0fcb6536d805"). InnerVolumeSpecName "kube-api-access-2pfmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:48.027628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.027591 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pfmd\" (UniqueName: \"kubernetes.io/projected/815ccca2-b75f-42fa-967a-0fcb6536d805-kube-api-access-2pfmd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.027628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.027626 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5054a9-5ceb-4bc5-b694-588624ecbb67-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.027628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.027638 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815ccca2-b75f-42fa-967a-0fcb6536d805-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.027872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.027650 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tfl\" (UniqueName: \"kubernetes.io/projected/5c5054a9-5ceb-4bc5-b694-588624ecbb67-kube-api-access-z5tfl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.499558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.499534 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:21:48.560313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560269 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560390 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560479 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560584 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:48.560689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560598 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560682 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560768 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560838 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560889 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560922 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\") pod \"58bee755-9e50-4cc3-b825-974fd9cea132\" (UID: \"58bee755-9e50-4cc3-b825-974fd9cea132\") " Oct 30 09:21:48.560961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.560941 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:48.561114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561041 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:48.561873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561238 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:48.561873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561346 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:48.561873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561818 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58bee755-9e50-4cc3-b825-974fd9cea132-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.561873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561854 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.561873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561869 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.562259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561882 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.562259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.561894 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.563643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.563617 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:21:48.563990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.563965 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:48.564141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.564016 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "kube-api-access-5qxch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:48.565090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.565068 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo" pod "58bee755-9e50-4cc3-b825-974fd9cea132" (UID: "58bee755-9e50-4cc3-b825-974fd9cea132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:21:48.605670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.605627 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" event={"ID":"e9ae024a-e18b-4c67-aa6d-156bea0ca507","Type":"ContainerStarted","Data":"bd880bfe09362facfb088783e5dc679ea535b79cc5efffaedd433c9cd7578f43"} Oct 30 09:21:48.605874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.605699 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" event={"ID":"e9ae024a-e18b-4c67-aa6d-156bea0ca507","Type":"ContainerStarted","Data":"ab5b0d4b51da7521397d8f8a1bd1f84dbc335233fa97603c48419ff9afda3f9e"} Oct 30 09:21:48.607975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.607423 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" Oct 30 09:21:48.607975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.607440 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v" event={"ID":"5c5054a9-5ceb-4bc5-b694-588624ecbb67","Type":"ContainerDied","Data":"c14b155223c47df4df0f51fe12d675c68d3fdfafe3764e311ea8666a67943e33"} Oct 30 09:21:48.607975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.607501 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14b155223c47df4df0f51fe12d675c68d3fdfafe3764e311ea8666a67943e33" Oct 30 09:21:48.609205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.609178 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" event={"ID":"815ccca2-b75f-42fa-967a-0fcb6536d805","Type":"ContainerDied","Data":"3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f"} Oct 30 09:21:48.609205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.609196 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6" Oct 30 09:21:48.609389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.609233 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3874ef26fc7df0beeceb66b945f364a547aee9353ab6b7401443b36730c6744f" Oct 30 09:21:48.611639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.611593 4920 generic.go:356] "Generic (PLEG): container finished" podID="58bee755-9e50-4cc3-b825-974fd9cea132" containerID="650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8" exitCode=0 Oct 30 09:21:48.611768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.611632 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerDied","Data":"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8"} Oct 30 09:21:48.611768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.611690 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" event={"ID":"58bee755-9e50-4cc3-b825-974fd9cea132","Type":"ContainerDied","Data":"11fc738d917699f1646ed39c2edcfee9fa71e7ae81e9e240a70d81cf5aae9217"} Oct 30 09:21:48.611768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.611720 4920 scope.go:122] "RemoveContainer" containerID="650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8" Oct 30 09:21:48.611928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.611643 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m" Oct 30 09:21:48.625087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.625065 4920 scope.go:122] "RemoveContainer" containerID="846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33" Oct 30 09:21:48.636666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.636646 4920 scope.go:122] "RemoveContainer" containerID="650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8" Oct 30 09:21:48.636996 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:48.636968 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8\": container with ID starting with 650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8 not found: ID does not exist" containerID="650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8" Oct 30 09:21:48.637079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.637010 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8"} err="failed to get container status \"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8\": rpc error: code = NotFound desc = could not find container \"650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8\": container with ID starting with 650de56c1cc700b1b55a8299096562f415883b0444762fc5f1a481343e7a81d8 not found: ID does not exist" Oct 30 09:21:48.637079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.637035 4920 scope.go:122] "RemoveContainer" containerID="846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33" Oct 30 09:21:48.637365 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:21:48.637342 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33\": container with ID starting with 846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33 not found: ID does not exist" containerID="846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33" Oct 30 09:21:48.637453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.637372 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33"} err="failed to get container status \"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33\": rpc error: code = NotFound desc = could not find container \"846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33\": container with ID starting with 846a941d80484fccb9af5211ec0e7ffcd54b429c7834403d6e65b9758e780b33 not found: ID does not exist" Oct 30 09:21:48.645958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.645911 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" podStartSLOduration=1.645899268 podStartE2EDuration="1.645899268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:48.642621038 +0000 UTC m=+1672.366048181" watchObservedRunningTime="2026-10-30 09:21:48.645899268 +0000 UTC m=+1672.369326411" Oct 30 09:21:48.663861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.663838 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58bee755-9e50-4cc3-b825-974fd9cea132-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.663861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.663859 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58bee755-9e50-4cc3-b825-974fd9cea132-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.664004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.663868 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qxch\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-kube-api-access-5qxch\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.664004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.663881 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58bee755-9e50-4cc3-b825-974fd9cea132-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:48.697245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.697202 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:21:48.703399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.703371 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca0df2-openshift-default-5f9dbf8c97-sfc2m"] Oct 30 09:21:48.961698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:48.961655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:49.305948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:49.305912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:49.622586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:49.622503 4920 generic.go:356] "Generic (PLEG): container finished" podID="e9ae024a-e18b-4c67-aa6d-156bea0ca507" containerID="bd880bfe09362facfb088783e5dc679ea535b79cc5efffaedd433c9cd7578f43" exitCode=0 Oct 30 09:21:49.622745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:49.622590 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" event={"ID":"e9ae024a-e18b-4c67-aa6d-156bea0ca507","Type":"ContainerDied","Data":"bd880bfe09362facfb088783e5dc679ea535b79cc5efffaedd433c9cd7578f43"} Oct 30 09:21:50.400016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.399982 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" path="/var/lib/kubelet/pods/58bee755-9e50-4cc3-b825-974fd9cea132/volumes" Oct 30 09:21:50.460037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.460005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:50.792437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.792409 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:50.899002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.898964 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\" (UniqueName: \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\") pod \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " Oct 30 09:21:50.899174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.899069 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\") pod \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " Oct 30 09:21:50.899334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.899312 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\") pod \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\" (UID: \"e9ae024a-e18b-4c67-aa6d-156bea0ca507\") " Oct 30 09:21:50.899735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.899681 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca" pod "e9ae024a-e18b-4c67-aa6d-156bea0ca507" (UID: "e9ae024a-e18b-4c67-aa6d-156bea0ca507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:50.900609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.900582 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:50.901744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.901717 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca" pod "e9ae024a-e18b-4c67-aa6d-156bea0ca507" (UID: "e9ae024a-e18b-4c67-aa6d-156bea0ca507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:50.901869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:50.901850 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb" pod "e9ae024a-e18b-4c67-aa6d-156bea0ca507" (UID: "e9ae024a-e18b-4c67-aa6d-156bea0ca507"). InnerVolumeSpecName "kube-api-access-x2cvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:51.001888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:51.001799 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cvb\" (UniqueName: \"kubernetes.io/projected/e9ae024a-e18b-4c67-aa6d-156bea0ca507-kube-api-access-x2cvb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:51.001888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:51.001828 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ae024a-e18b-4c67-aa6d-156bea0ca507-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:21:51.643960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:51.643926 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" Oct 30 09:21:51.644369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:51.643965 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59" event={"ID":"e9ae024a-e18b-4c67-aa6d-156bea0ca507","Type":"ContainerDied","Data":"ab5b0d4b51da7521397d8f8a1bd1f84dbc335233fa97603c48419ff9afda3f9e"} Oct 30 09:21:51.644369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:51.644011 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5b0d4b51da7521397d8f8a1bd1f84dbc335233fa97603c48419ff9afda3f9e" Oct 30 09:21:52.068598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:52.068559 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:53.637265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:53.637189 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:54.423746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:54.423336 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:54.516057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:54.516021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:55.363687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:55.363656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:56.958479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:56.958446 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:58.501034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:58.500999 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:21:59.464976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:59.464943 4920 scope.go:122] "RemoveContainer" containerID="e4501497189ce9ea427731324ba020725086c4eb2effef78cca3187e89cdd619" Oct 30 09:21:59.475679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:59.475657 4920 scope.go:122] "RemoveContainer" containerID="fbaf2d39532cf54f10827e0059f437ecc3179590f402d80332f3f801bd8312ce" Oct 30 09:21:59.485662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:59.485637 4920 scope.go:122] "RemoveContainer" containerID="f75706ea8ab0e38e95bcf4e55e018a9aebe2296bd2597dbfd1a4901ac1c22626" Oct 30 09:21:59.495445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:59.495418 4920 scope.go:122] "RemoveContainer" containerID="bcf2f564fb6decbc6a7e6bc07eed5de0b991ecee78bd813f7bfa7b80ad2341af" Oct 30 09:21:59.510902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:21:59.510877 4920 scope.go:122] "RemoveContainer" containerID="bd994a7b600bad8bafbe5cac8add36f1cc3613892294ac190fee5afc03df4e7c" Oct 30 09:22:00.074787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:00.074758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:00.381721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:00.381639 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:22:01.139973 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:22:01.139925 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:22:01.872236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:01.871295 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:03.475242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:03.474905 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:03.602248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:03.599312 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5cq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:22:03.602248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:03.599389 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:22:04.056923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.056820 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:04.125258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.124265 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:22:04.125258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.124356 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:22:04.181375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.181276 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:04.273864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.273707 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:04.468477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.468390 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl"] Oct 30 09:22:04.494657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:04.494608 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-hbfsl"] Oct 30 09:22:05.019124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.019089 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:05.127173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.127133 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:05.233959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.233920 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p"] Oct 30 09:22:05.241540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.241510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:05.245590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.245565 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c9de5-bb44p"] Oct 30 09:22:05.743104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.743054 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:05.842996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:05.842964 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:06.326391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:06.326354 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:06.564244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:06.564194 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172ab02a-26c0-44e3-b8e3-aecdd01b21f3" path="/var/lib/kubelet/pods/172ab02a-26c0-44e3-b8e3-aecdd01b21f3/volumes" Oct 30 09:22:06.565791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:06.565753 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8042310a-ae8e-4981-b067-4f646f8dbbed" path="/var/lib/kubelet/pods/8042310a-ae8e-4981-b067-4f646f8dbbed/volumes" Oct 30 09:22:06.634011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:06.633900 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:07.158328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:07.158289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:08.821002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:08.820967 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:08.925581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:08.925546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:10.391412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:10.391369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:10.496792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:10.496759 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:10.971258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:10.969781 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:11.799619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:11.799583 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:11.899443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:11.899402 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:12.162782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:12.162677 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:12.265959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:12.265913 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:12.690898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:12.690854 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:13.172909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:13.172879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:13.625919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:13.625880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:13.723691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:13.723618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:14.469514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:14.469477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:15.710033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:15.709991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:15.811753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:15.811721 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:16.451608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.451574 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:16.551698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.551665 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:16.651392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.651354 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:16.754179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.754096 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:16.760924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.760892 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:16.761563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761548 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e9ae024a-e18b-4c67-aa6d-156bea0ca507" containerName="revoke-keys" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761565 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ae024a-e18b-4c67-aa6d-156bea0ca507" containerName="revoke-keys" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761593 4920 cpu_manager.go:414] "containerMap: removing container" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761601 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761607 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5c5054a9-5ceb-4bc5-b694-588624ecbb67" containerName="revoke-keys" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761612 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5054a9-5ceb-4bc5-b694-588624ecbb67" containerName="revoke-keys" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761626 4920 cpu_manager.go:414] "containerMap: removing container" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:16.761635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761631 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761653 4920 cpu_manager.go:414] "containerMap: removing container" podUID="815ccca2-b75f-42fa-967a-0fcb6536d805" containerName="revoke-keys" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761658 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815ccca2-b75f-42fa-967a-0fcb6536d805" containerName="revoke-keys" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761761 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761768 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5054a9-5ceb-4bc5-b694-588624ecbb67" containerName="revoke-keys" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761780 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="815ccca2-b75f-42fa-967a-0fcb6536d805" containerName="revoke-keys" Oct 30 09:22:16.761870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.761788 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ae024a-e18b-4c67-aa6d-156bea0ca507" containerName="revoke-keys" Oct 30 09:22:16.767692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.767672 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:16.784203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.784175 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:16.790368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.790338 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8eada0cf\"" Oct 30 09:22:16.790553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.790496 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8eada0cf-dockercfg-xsnq6\"" Oct 30 09:22:16.979249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.945293 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:16.992252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:16.979777 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdhgl\" (UniqueName: \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.051303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.027425 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:17.073275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.052057 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bee755-9e50-4cc3-b825-974fd9cea132" containerName="istio-proxy" Oct 30 09:22:17.095198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.095161 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:17.095425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.095399 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.095567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.095544 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.095942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.095920 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdhgl\" (UniqueName: \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.096478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.096450 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.115273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.115242 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdhgl\" (UniqueName: \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.198137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.198097 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbwh\" (UniqueName: \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.198408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.198386 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.299538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.299501 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.299740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.299667 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbwh\" (UniqueName: \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.300324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.300294 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.319608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.319547 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbwh\" (UniqueName: \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.385479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.385441 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:17.421483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.418365 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:17.936182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.936145 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:17.997005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:17.996960 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:18.104760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:18.104710 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:18.153295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:18.153257 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" event={"ID":"21ce3fdf-f3db-4169-a1b9-db3bf44f3403","Type":"ContainerStarted","Data":"96310fb44e5f53c497ded440b977eb87b81f20f89f6ad5c094b0bee85415b1bd"} Oct 30 09:22:18.155527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:18.155492 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" event={"ID":"f8658c49-76e9-4f11-bceb-4873fb960ff4","Type":"ContainerStarted","Data":"19eb3fc1fc7ef0aed24888c6bbcfe9e59c4445cd6d15c38a88a8eca0f18e2b63"} Oct 30 09:22:18.219617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:18.219584 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:18.563834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:18.563797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:19.215242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.211577 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" event={"ID":"21ce3fdf-f3db-4169-a1b9-db3bf44f3403","Type":"ContainerStarted","Data":"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6"} Oct 30 09:22:19.233838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.232780 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:19.249289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.249251 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" event={"ID":"f8658c49-76e9-4f11-bceb-4873fb960ff4","Type":"ContainerStarted","Data":"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc"} Oct 30 09:22:19.250440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.250414 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:19.301239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.295450 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" podStartSLOduration=3.295429265 podStartE2EDuration="3.295429265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:19.271189134 +0000 UTC m=+1702.994616282" watchObservedRunningTime="2026-10-30 09:22:19.295429265 +0000 UTC m=+1703.018856410" Oct 30 09:22:19.319696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.315848 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" podStartSLOduration=3.315828458 podStartE2EDuration="3.315828458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:19.315721677 +0000 UTC m=+1703.039148811" watchObservedRunningTime="2026-10-30 09:22:19.315828458 +0000 UTC m=+1703.039255604" Oct 30 09:22:19.519158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.519059 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:19.669381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.669346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:19.716565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.716522 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v"] Oct 30 09:22:19.724643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.724598 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8d1106-8787v"] Oct 30 09:22:19.753643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:19.753610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:20.210237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:20.210184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:20.399939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:20.399902 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5054a9-5ceb-4bc5-b694-588624ecbb67" path="/var/lib/kubelet/pods/5c5054a9-5ceb-4bc5-b694-588624ecbb67/volumes" Oct 30 09:22:20.763150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:20.763112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:20.863635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:20.863600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:21.776286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:21.776212 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:23.345582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:23.345546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:24.841398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:24.841199 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:24.940013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:24.939978 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:25.268059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:25.267976 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:25.992303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:25.992249 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:26.089041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:26.088998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:26.361573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:26.361535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:26.862426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:26.862387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:27.615188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:27.615149 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:28.030143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.030069 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:22:28.035804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.035768 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.062756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.062729 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:22:28.065111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.065089 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8f40a3-openshift-default-dockercfg-99m9z\"" Oct 30 09:22:28.089373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.089341 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:28.177640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.177542 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.178004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.177977 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.178255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.178233 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7ql\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.178496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.178478 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.178727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.178710 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.179004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.178986 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.179165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.179149 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.179365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.179347 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.179548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.179531 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.284674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.284640 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.285467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.285391 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.285949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.285881 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.286129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.286109 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.286336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.286318 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.286915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.286769 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354468 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354611 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354666 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354697 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7ql\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354780 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.354866 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.377416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.371433 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.377416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.372108 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.377660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.377481 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.404486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.404449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:28.433594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.414727 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.439144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.438977 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7ql\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.441916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.441890 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\") pod \"e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.497109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.497074 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:28.662027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.661991 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:28.981754 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:22:28.981723 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107feebf_f523_41d5_8543_3601ad9e152f.slice/crio-b3c45322913a6a2a36f41d3de3f2f0db000d843fc10b389a391d540ef38b60c8 WatchSource:0}: Error finding container b3c45322913a6a2a36f41d3de3f2f0db000d843fc10b389a391d540ef38b60c8: Status 404 returned error can't find the container with id b3c45322913a6a2a36f41d3de3f2f0db000d843fc10b389a391d540ef38b60c8 Oct 30 09:22:28.984498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.984459 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:28.984576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.984546 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:28.984616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.984588 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:28.985556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:28.985539 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:22:29.451593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:29.451556 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerStarted","Data":"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f"} Oct 30 09:22:29.451751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:29.451621 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerStarted","Data":"b3c45322913a6a2a36f41d3de3f2f0db000d843fc10b389a391d540ef38b60c8"} Oct 30 09:22:29.508494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:29.508439 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podStartSLOduration=2.508422341 podStartE2EDuration="2.508422341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:29.49297842 +0000 UTC m=+1713.216405564" watchObservedRunningTime="2026-10-30 09:22:29.508422341 +0000 UTC m=+1713.231849555" Oct 30 09:22:29.663723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:29.663686 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:30.084937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:30.084905 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:30.667977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:30.667946 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:31.136717 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:22:31.136672 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:22:31.270163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.270137 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:31.270351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.270205 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:31.474642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.474555 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:31.475526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.475503 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:22:31.705779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.705748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:31.817341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.817298 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:31.915839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:31.915806 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:33.551813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:33.551778 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:33.636192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:33.636156 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:33.738531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:33.738502 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:33.837505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:33.837396 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:34.142862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.142771 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:22:34.198772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.198736 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:22:34.198970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.198863 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.232343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.225759 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f40a3\"" Oct 30 09:22:34.235127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.233268 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f40a3-dockercfg-pxg7m\"" Oct 30 09:22:34.284333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.284297 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.284549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.284505 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrt5\" (UniqueName: \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.386262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.386177 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.386472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.386337 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrt5\" (UniqueName: \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.386946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.386922 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.405171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.405097 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrt5\" (UniqueName: \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\") pod \"payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.564471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.563660 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:34.904173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:34.904133 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:22:35.527184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:35.527143 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" event={"ID":"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f","Type":"ContainerStarted","Data":"788d0e7b9f546e40e64056137f14c47d2653afc2e17ca23683bd21d1f7dfbf45"} Oct 30 09:22:35.527419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:35.527204 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" event={"ID":"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f","Type":"ContainerStarted","Data":"8a4b98ac3e27b6d8b99d8ff5308ca48d86f317a2ddd58b0c50789be0685df73e"} Oct 30 09:22:35.527419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:35.527270 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:35.566325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:35.566258 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" podStartSLOduration=2.5662062629999998 podStartE2EDuration="2.566206263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:35.565024657 +0000 UTC m=+1719.288451798" watchObservedRunningTime="2026-10-30 09:22:35.566206263 +0000 UTC m=+1719.289633407" Oct 30 09:22:35.749311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:35.749270 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:37.031715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:37.031678 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz"] Oct 30 09:22:37.045656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:37.045622 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139d508d-lhxlz"] Oct 30 09:22:37.336938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:37.336901 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:38.400699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:38.400664 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf13c69e-b17b-4a8b-b391-74813f8b3601" path="/var/lib/kubelet/pods/bf13c69e-b17b-4a8b-b391-74813f8b3601/volumes" Oct 30 09:22:38.744755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:38.744679 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:22:38.976591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:38.976542 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:40.752909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.752869 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj"] Oct 30 09:22:40.759724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.759689 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:40.760004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.759985 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:40.763395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.763366 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj"] Oct 30 09:22:40.774690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.774654 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:22:40.786469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.786435 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:22:40.786469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.786445 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:22:40.786661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.786573 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:22:40.949301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.949267 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpp6\" (UniqueName: \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:40.949481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.949383 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:40.949481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:40.949467 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.051285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.051250 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpp6\" (UniqueName: \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.051475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.051328 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.051475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.051396 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.052003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.051982 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.053769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.053749 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.069723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.069693 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpp6\" (UniqueName: \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\") pod \"maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.076713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.076689 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:41.385333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.385294 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:41.386447 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:22:41.386418 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe96d542_24a2_450e_acb3_620cf6959296.slice/crio-dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d WatchSource:0}: Error finding container dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d: Status 404 returned error can't find the container with id dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d Oct 30 09:22:41.391036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.391007 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj"] Oct 30 09:22:41.651757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.651676 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" event={"ID":"be96d542-24a2-450e-acb3-620cf6959296","Type":"ContainerStarted","Data":"dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d"} Oct 30 09:22:41.783594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.783559 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:22:41.788260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.788237 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.816678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.816646 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:22:41.823904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.823840 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-58fadf-openshift-default-dockercfg-6f8fb\"" Oct 30 09:22:41.854028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.853994 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:41.865908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.865867 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866010 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866064 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6sb\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866270 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866358 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866413 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866448 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866478 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.866609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.866564 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.986284 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.986730 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.986770 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.986797 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.986890 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.987012 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.987104 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.987159 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6sb\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.987865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.987396 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.994320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.988088 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.994320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.988739 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.994320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.988996 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:41.994320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:41.989240 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.027793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.027732 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.028210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.028154 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.038279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.034631 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.059991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.059954 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.076016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.075977 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6sb\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\") pod \"e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.125170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.125134 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:42.227391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.227312 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:42.568703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.567878 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:22:42.578311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.576342 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:42.578311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.576428 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:42.578311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.576475 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:42.663881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.663852 4920 generic.go:356] "Generic (PLEG): container finished" podID="be96d542-24a2-450e-acb3-620cf6959296" containerID="29324f22eda4f9edeeaec4660291a465ec47cba407388de58588173d878fcb77" exitCode=0 Oct 30 09:22:42.664013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.663932 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" event={"ID":"be96d542-24a2-450e-acb3-620cf6959296","Type":"ContainerDied","Data":"29324f22eda4f9edeeaec4660291a465ec47cba407388de58588173d878fcb77"} Oct 30 09:22:42.665927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.665900 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerStarted","Data":"285ee522f1efa87982c0e6f303dffa52fbdfc8206104fbbd16731ade030caace"} Oct 30 09:22:42.666043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.665948 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerStarted","Data":"11b7e6c10e792a2fc58ae673db711ceb3d5b4b2fa9ae392d93212bf969c93840"} Oct 30 09:22:42.744518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.744477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:42.745646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:42.745585 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podStartSLOduration=1.745568488 podStartE2EDuration="1.745568488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:42.738791705 +0000 UTC m=+1726.462218851" watchObservedRunningTime="2026-10-30 09:22:42.745568488 +0000 UTC m=+1726.468995631" Oct 30 09:22:43.126179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.126131 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:43.132007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.131978 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:43.674522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.674486 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:43.675591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.675571 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:22:43.894892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.894864 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:43.971012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.970919 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\") pod \"be96d542-24a2-450e-acb3-620cf6959296\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " Oct 30 09:22:43.971290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.971269 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\") pod \"be96d542-24a2-450e-acb3-620cf6959296\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " Oct 30 09:22:43.971394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.971349 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\" (UniqueName: \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\") pod \"be96d542-24a2-450e-acb3-620cf6959296\" (UID: \"be96d542-24a2-450e-acb3-620cf6959296\") " Oct 30 09:22:43.973888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.973589 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca" pod "be96d542-24a2-450e-acb3-620cf6959296" (UID: "be96d542-24a2-450e-acb3-620cf6959296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:43.985357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.985322 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca" pod "be96d542-24a2-450e-acb3-620cf6959296" (UID: "be96d542-24a2-450e-acb3-620cf6959296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:43.986406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:43.986296 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6" pod "be96d542-24a2-450e-acb3-620cf6959296" (UID: "be96d542-24a2-450e-acb3-620cf6959296"). InnerVolumeSpecName "kube-api-access-zvpp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:44.075292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.075254 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be96d542-24a2-450e-acb3-620cf6959296-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:44.075292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.075293 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpp6\" (UniqueName: \"kubernetes.io/projected/be96d542-24a2-450e-acb3-620cf6959296-kube-api-access-zvpp6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:44.075519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.075312 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be96d542-24a2-450e-acb3-620cf6959296-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:44.588170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.588136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:44.687008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.686913 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" Oct 30 09:22:44.687178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.687057 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj" event={"ID":"be96d542-24a2-450e-acb3-620cf6959296","Type":"ContainerDied","Data":"dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d"} Oct 30 09:22:44.687178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:44.687125 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe34864d1fbdab667dd83cdf040005d3467d85444346526347feb7f2996f83d" Oct 30 09:22:45.112540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:45.112499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:45.221280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:45.221244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:45.977647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:45.977603 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:46.083730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:46.083695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:46.216563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:46.216524 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:46.446081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:46.446045 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:46.539948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:46.539918 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:22:47.357890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:47.357859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:47.457469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:47.457434 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:48.261561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:48.261527 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:48.364280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:48.364235 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:48.472577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:48.472540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:49.366753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:49.366716 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:50.002983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:50.002948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:51.206032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.205996 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:51.242691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.242648 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:51.338378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.338338 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:51.497968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.497871 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:51.499908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.499877 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:22:51.504431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.501145 4920 cpu_manager.go:414] "containerMap: removing container" podUID="be96d542-24a2-450e-acb3-620cf6959296" containerName="revoke-keys" Oct 30 09:22:51.504714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.504694 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be96d542-24a2-450e-acb3-620cf6959296" containerName="revoke-keys" Oct 30 09:22:51.506984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.506959 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="be96d542-24a2-450e-acb3-620cf6959296" containerName="revoke-keys" Oct 30 09:22:51.513775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.513745 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.516721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.516694 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:22:51.529149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.529118 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8de308-openshift-default-dockercfg-9xrhg\"" Oct 30 09:22:51.643175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643142 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643304 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9dh\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643370 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643410 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643436 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643584 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643616 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643725 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.643836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.643820 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.718186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.718152 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:51.744752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.744714 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.744926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.744815 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9dh\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.744926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.744847 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.744926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.744886 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.744926 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745016 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745043 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745112 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745204 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.745897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745867 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.746016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.745941 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.746016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.746000 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.746137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.746037 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.746137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.746040 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.760629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.760559 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.761694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.761669 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.767099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.767071 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9dh\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.767201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.767142 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:51.838091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:51.838049 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:52.083785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.064396 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4"] Oct 30 09:22:52.114860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.111402 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4c065a-8fmt4"] Oct 30 09:22:52.356012 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:22:52.355973 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb725a669_6638_4958_a8ca_e141e8999180.slice/crio-7a3ddfbd62b19201e2c264607e3e61acedd820ea0407992f1f44a13bcdb0130d WatchSource:0}: Error finding container 7a3ddfbd62b19201e2c264607e3e61acedd820ea0407992f1f44a13bcdb0130d: Status 404 returned error can't find the container with id 7a3ddfbd62b19201e2c264607e3e61acedd820ea0407992f1f44a13bcdb0130d Oct 30 09:22:52.359163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.359127 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:52.359305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.359200 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:52.359305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.359277 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:52.360552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.360530 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:22:52.401466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:52.401433 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1710a2-5f7c-4d06-8f07-4269b0d14734" path="/var/lib/kubelet/pods/9d1710a2-5f7c-4d06-8f07-4269b0d14734/volumes" Oct 30 09:22:53.027663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.027602 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" event={"ID":"b725a669-6638-4958-a8ca-e141e8999180","Type":"ContainerStarted","Data":"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4"} Oct 30 09:22:53.027891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.027685 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" event={"ID":"b725a669-6638-4958-a8ca-e141e8999180","Type":"ContainerStarted","Data":"7a3ddfbd62b19201e2c264607e3e61acedd820ea0407992f1f44a13bcdb0130d"} Oct 30 09:22:53.523239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.518848 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:53.838649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.838613 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:53.840363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.840336 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:22:53.840482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:53.840397 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 09:22:54.839231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:54.839191 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:22:54.839755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:54.839256 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 09:22:55.284180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:55.284148 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:55.839262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:55.839039 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:22:55.839262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:55.839103 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 09:22:56.216301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.215299 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" podUID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" containerName="payload-processing" containerID="cri-o://798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6" gracePeriod=25 Oct 30 09:22:56.243021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.242970 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" podUID="f8658c49-76e9-4f11-bceb-4873fb960ff4" containerName="payload-pre-processing" containerID="cri-o://c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc" gracePeriod=25 Oct 30 09:22:56.647534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.647503 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:56.671617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.671597 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:56.674338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.674313 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\" (UniqueName: \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\") pod \"f8658c49-76e9-4f11-bceb-4873fb960ff4\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " Oct 30 09:22:56.674603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.674582 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\") pod \"f8658c49-76e9-4f11-bceb-4873fb960ff4\" (UID: \"f8658c49-76e9-4f11-bceb-4873fb960ff4\") " Oct 30 09:22:56.674945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.674909 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume" pod "f8658c49-76e9-4f11-bceb-4873fb960ff4" (UID: "f8658c49-76e9-4f11-bceb-4873fb960ff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:56.675524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.675504 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8658c49-76e9-4f11-bceb-4873fb960ff4-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:56.676837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.676816 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl" pod "f8658c49-76e9-4f11-bceb-4873fb960ff4" (UID: "f8658c49-76e9-4f11-bceb-4873fb960ff4"). InnerVolumeSpecName "kube-api-access-rdhgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:56.693828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.693781 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podStartSLOduration=5.693765629 podStartE2EDuration="5.693765629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:53.092121751 +0000 UTC m=+1736.815548894" watchObservedRunningTime="2026-10-30 09:22:56.693765629 +0000 UTC m=+1740.417192771" Oct 30 09:22:56.776666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.776626 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\" (UniqueName: \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\") pod \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " Oct 30 09:22:56.776986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.776962 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\") pod \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\" (UID: \"21ce3fdf-f3db-4169-a1b9-db3bf44f3403\") " Oct 30 09:22:56.777422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.777375 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume" pod "21ce3fdf-f3db-4169-a1b9-db3bf44f3403" (UID: "21ce3fdf-f3db-4169-a1b9-db3bf44f3403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:56.777981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.777953 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:56.778079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.777987 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdhgl\" (UniqueName: \"kubernetes.io/projected/f8658c49-76e9-4f11-bceb-4873fb960ff4-kube-api-access-rdhgl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:56.779508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.779478 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh" pod "21ce3fdf-f3db-4169-a1b9-db3bf44f3403" (UID: "21ce3fdf-f3db-4169-a1b9-db3bf44f3403"). InnerVolumeSpecName "kube-api-access-qdbwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:56.839301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.839207 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:22:56.839774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.839744 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 09:22:56.879802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.879766 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbwh\" (UniqueName: \"kubernetes.io/projected/21ce3fdf-f3db-4169-a1b9-db3bf44f3403-kube-api-access-qdbwh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:22:56.982056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:56.982017 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:57.086006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.085961 4920 generic.go:356] "Generic (PLEG): container finished" podID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" containerID="798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6" exitCode=0 Oct 30 09:22:57.086428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.086395 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" event={"ID":"21ce3fdf-f3db-4169-a1b9-db3bf44f3403","Type":"ContainerDied","Data":"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6"} Oct 30 09:22:57.086657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.086631 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" event={"ID":"21ce3fdf-f3db-4169-a1b9-db3bf44f3403","Type":"ContainerDied","Data":"96310fb44e5f53c497ded440b977eb87b81f20f89f6ad5c094b0bee85415b1bd"} Oct 30 09:22:57.086812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.086798 4920 scope.go:122] "RemoveContainer" containerID="798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6" Oct 30 09:22:57.087063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.087048 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4" Oct 30 09:22:57.116538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.116505 4920 generic.go:356] "Generic (PLEG): container finished" podID="f8658c49-76e9-4f11-bceb-4873fb960ff4" containerID="c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc" exitCode=0 Oct 30 09:22:57.134371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.119191 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" Oct 30 09:22:57.134371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.120685 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" event={"ID":"f8658c49-76e9-4f11-bceb-4873fb960ff4","Type":"ContainerDied","Data":"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc"} Oct 30 09:22:57.134371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.120757 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv" event={"ID":"f8658c49-76e9-4f11-bceb-4873fb960ff4","Type":"ContainerDied","Data":"19eb3fc1fc7ef0aed24888c6bbcfe9e59c4445cd6d15c38a88a8eca0f18e2b63"} Oct 30 09:22:57.167110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.167078 4920 scope.go:122] "RemoveContainer" containerID="798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6" Oct 30 09:22:57.167573 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:22:57.167545 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6\": container with ID starting with 798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6 not found: ID does not exist" containerID="798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6" Oct 30 09:22:57.167672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.167585 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6"} err="failed to get container status \"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6\": rpc error: code = NotFound desc = could not find container \"798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6\": container with ID starting with 798e05ae207c396106b2b7d504f9e92fff5c5ee7e34195a718b1f8adba019cc6 not found: ID does not exist" Oct 30 09:22:57.167672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.167611 4920 scope.go:122] "RemoveContainer" containerID="c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc" Oct 30 09:22:57.180856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.180832 4920 scope.go:122] "RemoveContainer" containerID="c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc" Oct 30 09:22:57.181190 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:22:57.181168 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc\": container with ID starting with c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc not found: ID does not exist" containerID="c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc" Oct 30 09:22:57.181317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.181200 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc"} err="failed to get container status \"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc\": rpc error: code = NotFound desc = could not find container \"c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc\": container with ID starting with c33182a68cd81270a266e70b0e95191f52d6a79a0e84528878eb7eb54599eacc not found: ID does not exist" Oct 30 09:22:57.181386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.181362 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:57.189493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.189467 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-fh7m4"] Oct 30 09:22:57.234971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.234923 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:57.243244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.243197 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-jn6vv"] Oct 30 09:22:57.838502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.838465 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:22:57.838688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:57.838544 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 09:22:58.274760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.274614 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:58.369983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.369953 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:58.455268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.451616 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" path="/var/lib/kubelet/pods/21ce3fdf-f3db-4169-a1b9-db3bf44f3403/volumes" Oct 30 09:22:58.455268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.452272 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8658c49-76e9-4f11-bceb-4873fb960ff4" path="/var/lib/kubelet/pods/f8658c49-76e9-4f11-bceb-4873fb960ff4/volumes" Oct 30 09:22:58.473643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.472474 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:58.843058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.843027 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:58.843407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.843384 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:58.844107 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.844089 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:22:58.872452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:58.872425 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:22:59.584543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:59.584510 4920 scope.go:122] "RemoveContainer" containerID="4299d2671067fa3dc7a8934981960340e7e0a545230f9d43a6bca2a48d280282" Oct 30 09:22:59.596540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:59.596520 4920 scope.go:122] "RemoveContainer" containerID="64ff0148c6bcabd29ee2580ecdf1499380fe888b7006961a178999edfdd91553" Oct 30 09:22:59.607536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:22:59.607505 4920 scope.go:122] "RemoveContainer" containerID="9956a2e5894b9f9acdab5b9546899a3cde61387ba9fa8de2195189d642cb2837" Oct 30 09:23:00.162046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.162015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:00.363893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.357903 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.921016 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922049 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f8658c49-76e9-4f11-bceb-4873fb960ff4" containerName="payload-pre-processing" Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922070 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8658c49-76e9-4f11-bceb-4873fb960ff4" containerName="payload-pre-processing" Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922135 4920 cpu_manager.go:414] "containerMap: removing container" podUID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" containerName="payload-processing" Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922143 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" containerName="payload-processing" Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922379 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ce3fdf-f3db-4169-a1b9-db3bf44f3403" containerName="payload-processing" Oct 30 09:23:00.923252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:00.922401 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8658c49-76e9-4f11-bceb-4873fb960ff4" containerName="payload-pre-processing" Oct 30 09:23:01.005242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.004338 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.039268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.034413 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8de308-dockercfg-gdh57\"" Oct 30 09:23:01.039268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.034525 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8de308\"" Oct 30 09:23:01.053407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.053369 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:23:01.069239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.065786 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zs8\" (UniqueName: \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.069239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.066037 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.137177 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:23:01.137137 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:23:01.167514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.167470 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zs8\" (UniqueName: \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.167697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.167648 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.168442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.168420 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.186360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.186295 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zs8\" (UniqueName: \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\") pod \"payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.347564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.347527 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:01.495572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.495476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:01.671895 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:23:01.671866 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86208ffd_5a4f_4fe5_8171_bd9d3320f053.slice/crio-2fdc80329303454e512709f0a6960c91782de8a89a1fb8e7f1f39e76e2af3f88 WatchSource:0}: Error finding container 2fdc80329303454e512709f0a6960c91782de8a89a1fb8e7f1f39e76e2af3f88: Status 404 returned error can't find the container with id 2fdc80329303454e512709f0a6960c91782de8a89a1fb8e7f1f39e76e2af3f88 Oct 30 09:23:01.675484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:01.675461 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:23:02.215957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:02.215919 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" event={"ID":"86208ffd-5a4f-4fe5-8171-bd9d3320f053","Type":"ContainerStarted","Data":"2fdc80329303454e512709f0a6960c91782de8a89a1fb8e7f1f39e76e2af3f88"} Oct 30 09:23:03.255641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:03.255600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:03.353034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:03.352997 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:03.525312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:03.525102 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:04.244591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:04.244495 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" event={"ID":"86208ffd-5a4f-4fe5-8171-bd9d3320f053","Type":"ContainerStarted","Data":"e6853802f6cc324d6281f7e32504ed245ac8237109c101660c234843251fdafd"} Oct 30 09:23:04.244727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:04.244589 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:04.287651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:04.287591 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" podStartSLOduration=2.1218971 podStartE2EDuration="4.28757356s" totalImagesPullingTime="2.16567646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:04.282884721 +0000 UTC m=+1748.006311868" watchObservedRunningTime="2026-10-30 09:23:04.28757356 +0000 UTC m=+1748.011000705" Oct 30 09:23:05.036921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:05.036880 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt"] Oct 30 09:23:05.044423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:05.044393 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df24201-lsnwt"] Oct 30 09:23:05.286836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:05.286804 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:06.046774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:06.046729 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm"] Oct 30 09:23:06.053106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:06.053076 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45edd00a-84tfm"] Oct 30 09:23:06.434768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:06.434642 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d670a2-0b5e-4576-a954-4c0db0b92236" path="/var/lib/kubelet/pods/02d670a2-0b5e-4576-a954-4c0db0b92236/volumes" Oct 30 09:23:06.435364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:06.435343 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e980f5ac-1335-4aa9-85d6-2218c3ec12b3" path="/var/lib/kubelet/pods/e980f5ac-1335-4aa9-85d6-2218c3ec12b3/volumes" Oct 30 09:23:06.991982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:06.991946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:08.681378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:08.681293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:10.108515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:10.108409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:10.955415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:10.955382 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:11.092802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:11.092774 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:11.141651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:11.141611 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj"] Oct 30 09:23:11.164718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:11.164666 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8eada0cf-5kgwj"] Oct 30 09:23:11.276691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:11.276598 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:11.630237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:11.629516 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:12.403906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.403867 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be96d542-24a2-450e-acb3-620cf6959296" path="/var/lib/kubelet/pods/be96d542-24a2-450e-acb3-620cf6959296/volumes" Oct 30 09:23:12.488653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.488608 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:12.502834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.502796 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:23:12.646191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.646155 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:12.740535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.740443 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:12.837660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:12.837624 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:13.090947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.087428 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:23:13.096933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.096901 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.107288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.107256 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:23:13.148104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.147602 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c44644\"" Oct 30 09:23:13.148104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.147672 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c44644-dockercfg-kmx54\"" Oct 30 09:23:13.218087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.217655 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.218087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.217823 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gr5\" (UniqueName: \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.372297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.341277 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.372297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.341445 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gr5\" (UniqueName: \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.372297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.342738 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.387533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.387462 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gr5\" (UniqueName: \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\") pod \"payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.481737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.481433 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:13.883497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.883466 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:13.922368 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:23:13.922336 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7151869_fb12_484e_8ce0_eb25159807a7.slice/crio-4cab636b7901a2695dfed136a5fc872d0b9a829c44e3b710e0b356dcc16ce8d1 WatchSource:0}: Error finding container 4cab636b7901a2695dfed136a5fc872d0b9a829c44e3b710e0b356dcc16ce8d1: Status 404 returned error can't find the container with id 4cab636b7901a2695dfed136a5fc872d0b9a829c44e3b710e0b356dcc16ce8d1 Oct 30 09:23:13.923920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:13.923889 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:23:14.577237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:14.576630 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" event={"ID":"a7151869-fb12-484e-8ce0-eb25159807a7","Type":"ContainerStarted","Data":"18a904d52f587556899674dc6e14dc59c5116afbc4cb0b7501f75f21d70575a5"} Oct 30 09:23:14.577237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:14.576704 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" event={"ID":"a7151869-fb12-484e-8ce0-eb25159807a7","Type":"ContainerStarted","Data":"4cab636b7901a2695dfed136a5fc872d0b9a829c44e3b710e0b356dcc16ce8d1"} Oct 30 09:23:14.610310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:14.577924 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:14.691239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:14.690595 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" podStartSLOduration=1.690573014 podStartE2EDuration="1.690573014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:14.646751483 +0000 UTC m=+1758.370178653" watchObservedRunningTime="2026-10-30 09:23:14.690573014 +0000 UTC m=+1758.414000158" Oct 30 09:23:15.255631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.255599 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:23:15.511568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.511481 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:15.608193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.608096 4920 generic.go:356] "Generic (PLEG): container finished" podID="107feebf-f523-41d5-8543-3601ad9e152f" containerID="20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f" exitCode=0 Oct 30 09:23:15.608706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.608423 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerDied","Data":"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f"} Oct 30 09:23:15.609030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.609010 4920 scope.go:122] "RemoveContainer" containerID="20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f" Oct 30 09:23:15.610907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.610879 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:15.611083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.611061 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:15.611275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:15.611208 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:16.072277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.072230 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs"] Oct 30 09:23:16.080547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.080515 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1f51fb8-6sxjs"] Oct 30 09:23:16.088606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.088575 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7"] Oct 30 09:23:16.093607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.093578 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b316fb13-pc7l7"] Oct 30 09:23:16.429431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.426655 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eeb6682-2b23-4593-ad17-ac8cf31f3057" path="/var/lib/kubelet/pods/3eeb6682-2b23-4593-ad17-ac8cf31f3057/volumes" Oct 30 09:23:16.429431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.427422 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80620d7-2367-44fd-9174-999c48e3c968" path="/var/lib/kubelet/pods/d80620d7-2367-44fd-9174-999c48e3c968/volumes" Oct 30 09:23:16.474658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.474625 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:23:16.637918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.634312 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerStarted","Data":"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b"} Oct 30 09:23:16.663170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.663137 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:23:16.664514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.664489 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:16.664634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.664535 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:16.690175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:16.690110 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:17.323241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.322804 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:17.515244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.505295 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" podUID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" containerName="payload-processing" containerID="cri-o://788d0e7b9f546e40e64056137f14c47d2653afc2e17ca23683bd21d1f7dfbf45" gracePeriod=25 Oct 30 09:23:17.646797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.646767 4920 generic.go:356] "Generic (PLEG): container finished" podID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" containerID="788d0e7b9f546e40e64056137f14c47d2653afc2e17ca23683bd21d1f7dfbf45" exitCode=0 Oct 30 09:23:17.647147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.646844 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" event={"ID":"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f","Type":"ContainerDied","Data":"788d0e7b9f546e40e64056137f14c47d2653afc2e17ca23683bd21d1f7dfbf45"} Oct 30 09:23:17.663245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.663193 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:17.663385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.663266 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:17.851551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.851517 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:23:17.958700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.958599 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\" (UniqueName: \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\") pod \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " Oct 30 09:23:17.958700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.958669 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\") pod \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\" (UID: \"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f\") " Oct 30 09:23:17.959119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.959087 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume" pod "e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" (UID: "e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:23:17.959362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.959341 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:23:17.961527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:17.961499 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5" pod "e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" (UID: "e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f"). InnerVolumeSpecName "kube-api-access-7xrt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:18.025155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.022180 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:18.061191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.061159 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrt5\" (UniqueName: \"kubernetes.io/projected/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f-kube-api-access-7xrt5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:23:18.486010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.485971 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:23:18.487111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.487075 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" containerName="payload-processing" Oct 30 09:23:18.487111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.487105 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" containerName="payload-processing" Oct 30 09:23:18.487399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.487380 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" containerName="payload-processing" Oct 30 09:23:18.493184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.492972 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.531237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.525373 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cbe974-openshift-default-dockercfg-bmmrf\"" Oct 30 09:23:18.551269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.550883 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:23:18.572183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.572147 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:18.657684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.657649 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" Oct 30 09:23:18.657684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.657675 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp" event={"ID":"e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f","Type":"ContainerDied","Data":"8a4b98ac3e27b6d8b99d8ff5308ca48d86f317a2ddd58b0c50789be0685df73e"} Oct 30 09:23:18.658192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.657742 4920 scope.go:122] "RemoveContainer" containerID="788d0e7b9f546e40e64056137f14c47d2653afc2e17ca23683bd21d1f7dfbf45" Oct 30 09:23:18.662855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.662829 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:18.662983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.662885 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674394 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hsnv\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674473 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674527 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674663 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674760 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674834 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674882 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674917 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.676680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.674942 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.730318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.727252 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780168 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780244 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780274 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780551 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hsnv\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780609 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780661 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780731 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780810 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.781434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.780847 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.803350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.785709 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.803350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.786304 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.803350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.786687 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.803350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.787163 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.803350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.791347 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.817061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.816532 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.857268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.820010 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:23:18.869284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.868983 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.875259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.869506 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.876093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.876062 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hsnv\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\") pod \"e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:18.901366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:18.901329 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-575d669ff6-pjwxp"] Oct 30 09:23:19.153341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.153299 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:19.337879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.337840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:19.538510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.538477 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:23:19.541512 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:23:19.540905 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a73090_2cee_41c2_8873_492b7bfd297d.slice/crio-9a8502bf5b6d6d2a542b0793a366e1cbc5e62a99f9b9ff12cbc9eb57eecc0ea7 WatchSource:0}: Error finding container 9a8502bf5b6d6d2a542b0793a366e1cbc5e62a99f9b9ff12cbc9eb57eecc0ea7: Status 404 returned error can't find the container with id 9a8502bf5b6d6d2a542b0793a366e1cbc5e62a99f9b9ff12cbc9eb57eecc0ea7 Oct 30 09:23:19.543817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.543784 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:19.543909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.543885 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:19.543979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.543946 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:19.662885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.662850 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:19.663285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.662904 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:19.670772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.670738 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerStarted","Data":"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026"} Oct 30 09:23:19.670946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.670800 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerStarted","Data":"9a8502bf5b6d6d2a542b0793a366e1cbc5e62a99f9b9ff12cbc9eb57eecc0ea7"} Oct 30 09:23:19.738264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:19.735709 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podStartSLOduration=1.735687266 podStartE2EDuration="1.735687266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:19.732570522 +0000 UTC m=+1763.455997665" watchObservedRunningTime="2026-10-30 09:23:19.735687266 +0000 UTC m=+1763.459114409" Oct 30 09:23:20.153559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.153513 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:20.159248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.159206 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:20.423587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.423494 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f" path="/var/lib/kubelet/pods/e178e1b1-ed5b-4e8d-b494-bf1bb3469a3f/volumes" Oct 30 09:23:20.663408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.663374 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:20.663891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.663426 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:20.679982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.679892 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:20.680767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:20.680746 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:23:21.620241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:21.605230 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:21.664436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:21.662517 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 09:23:21.664436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:21.662577 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 09:23:22.131944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:22.131879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:22.305027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:22.304993 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:22.666982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:22.666944 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:23:22.667489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:22.667475 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:23:22.668110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:22.668089 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:23:23.479820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:23.479786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:23.896508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:23.896475 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:23.992866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:23.992830 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:25.615203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:25.615167 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:26.636992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:26.636958 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:23:27.279923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:27.279888 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:28.813129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:28.813040 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:29.036742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:29.036708 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v"] Oct 30 09:23:29.051146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:29.051110 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97930d8f-p9l7v"] Oct 30 09:23:29.909527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:29.909461 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:23:29.910251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:29.910197 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:30.176288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.160498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:30.386241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.368209 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:30.654632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.654573 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2ceac0-04fd-4732-a294-aeeebcde7fff" path="/var/lib/kubelet/pods/dd2ceac0-04fd-4732-a294-aeeebcde7fff/volumes" Oct 30 09:23:30.655181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.655160 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:23:30.669257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.666511 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:23:30.669257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.666631 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.682818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.682776 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.683163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.683120 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmvs\" (UniqueName: \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.694166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.694115 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f40a3\"" Oct 30 09:23:30.694685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.694649 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f40a3-dockercfg-nn5qf\"" Oct 30 09:23:30.785027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.784990 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.785240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.785209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmvs\" (UniqueName: \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.785844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.785819 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.811503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.811471 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmvs\" (UniqueName: \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\") pod \"payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:30.838979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.838941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:30.936884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.936801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:30.992236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:30.989194 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:31.139335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.132805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:31.338268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.336615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:31.350847 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:23:31.350061 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:23:31.554734 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:23:31.554693 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31fd1df_5a8a_4dcd_940d_17aae905fc95.slice/crio-fba4a47901500fb6bd2ba18f100403e52831004b0c5b1dda11f266515b98287d WatchSource:0}: Error finding container fba4a47901500fb6bd2ba18f100403e52831004b0c5b1dda11f266515b98287d: Status 404 returned error can't find the container with id fba4a47901500fb6bd2ba18f100403e52831004b0c5b1dda11f266515b98287d Oct 30 09:23:31.556295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.556271 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:23:31.792998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.792961 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:31.835261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.835119 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" event={"ID":"b31fd1df-5a8a-4dcd-940d-17aae905fc95","Type":"ContainerStarted","Data":"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3"} Oct 30 09:23:31.835261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.835190 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" event={"ID":"b31fd1df-5a8a-4dcd-940d-17aae905fc95","Type":"ContainerStarted","Data":"fba4a47901500fb6bd2ba18f100403e52831004b0c5b1dda11f266515b98287d"} Oct 30 09:23:31.835261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.835239 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:31.884925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:31.884872 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" podStartSLOduration=1.884856329 podStartE2EDuration="1.884856329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:31.879270828 +0000 UTC m=+1775.602697972" watchObservedRunningTime="2026-10-30 09:23:31.884856329 +0000 UTC m=+1775.608283736" Oct 30 09:23:32.056936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.056891 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g"] Oct 30 09:23:32.061905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.061875 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9"] Oct 30 09:23:32.068822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.068787 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a05867-wnp7g"] Oct 30 09:23:32.073784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.073764 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87edf0d-tmxn9"] Oct 30 09:23:32.182936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.182805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:32.484292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.483446 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67df86d7-bff6-46df-a181-d2a7f18af62c" path="/var/lib/kubelet/pods/67df86d7-bff6-46df-a181-d2a7f18af62c/volumes" Oct 30 09:23:32.484504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.484482 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68386fd-f97d-42a8-891f-7631f7bec75f" path="/var/lib/kubelet/pods/b68386fd-f97d-42a8-891f-7631f7bec75f/volumes" Oct 30 09:23:32.791451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:32.791412 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:33.676146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:33.676109 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:35.173123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:35.173084 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:35.289385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:35.289352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:35.389753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:35.389717 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:36.036910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:36.036871 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf"] Oct 30 09:23:36.044076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:36.044044 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d234355c-6d5lf"] Oct 30 09:23:36.433415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:36.429504 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff673e7d-735f-46ba-afca-4e10cdb88057" path="/var/lib/kubelet/pods/ff673e7d-735f-46ba-afca-4e10cdb88057/volumes" Oct 30 09:23:36.736142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:36.736031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:36.882977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:36.882943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:37.025293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:37.024674 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:37.520892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:37.520853 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:38.913013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:38.912971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:39.008275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.008241 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:39.369200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.369167 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:39.438242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.407339 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:23:39.501235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.491201 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.554658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.520462 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9371b7-dockercfg-smg8n\"" Oct 30 09:23:39.554658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.520537 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9371b7\"" Oct 30 09:23:39.554658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.543702 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:23:39.689070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.688971 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqpp\" (UniqueName: \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.689427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.689397 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.790507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.790472 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.790685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.790635 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqpp\" (UniqueName: \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.791441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.791418 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.812832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.812795 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqpp\" (UniqueName: \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\") pod \"payload-processing-e2e-api-b-9371b7-789f894498-rb2gj\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:39.882423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:39.882391 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:40.213523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:40.213431 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:23:40.218579 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:23:40.217898 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7a641a_6b65_4292_afb6_96a94411ca49.slice/crio-87ccf29eb60d939dc4ea3bd847f144aa8b43a990b931bc5f3abaf4a99b919b35 WatchSource:0}: Error finding container 87ccf29eb60d939dc4ea3bd847f144aa8b43a990b931bc5f3abaf4a99b919b35: Status 404 returned error can't find the container with id 87ccf29eb60d939dc4ea3bd847f144aa8b43a990b931bc5f3abaf4a99b919b35 Oct 30 09:23:40.932696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:40.932656 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" event={"ID":"bc7a641a-6b65-4292-afb6-96a94411ca49","Type":"ContainerStarted","Data":"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696"} Oct 30 09:23:40.932874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:40.932719 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" event={"ID":"bc7a641a-6b65-4292-afb6-96a94411ca49","Type":"ContainerStarted","Data":"87ccf29eb60d939dc4ea3bd847f144aa8b43a990b931bc5f3abaf4a99b919b35"} Oct 30 09:23:40.932874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:40.932755 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:40.970676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:40.970620 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" podStartSLOduration=1.970607111 podStartE2EDuration="1.970607111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:40.970285374 +0000 UTC m=+1784.693712518" watchObservedRunningTime="2026-10-30 09:23:40.970607111 +0000 UTC m=+1784.694034254" Oct 30 09:23:41.651673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:41.651635 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:41.740544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:41.740509 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:41.859254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:41.858748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:42.849769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.849741 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:23:42.958465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.958429 4920 generic.go:356] "Generic (PLEG): container finished" podID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerID="285ee522f1efa87982c0e6f303dffa52fbdfc8206104fbbd16731ade030caace" exitCode=0 Oct 30 09:23:42.958649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.958505 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerDied","Data":"285ee522f1efa87982c0e6f303dffa52fbdfc8206104fbbd16731ade030caace"} Oct 30 09:23:42.959033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.959014 4920 scope.go:122] "RemoveContainer" containerID="285ee522f1efa87982c0e6f303dffa52fbdfc8206104fbbd16731ade030caace" Oct 30 09:23:42.961087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.961051 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:42.961186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.961142 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:42.961268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:42.961184 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:43.049662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:43.048400 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k"] Oct 30 09:23:43.064573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:43.064524 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08497a89-dhd5k"] Oct 30 09:23:43.400885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:43.400840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:43.674769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:43.674679 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:23:43.974929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:43.974822 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerStarted","Data":"bda354e62e5922babb734bf87526ffaa8cdd41ff466284f07734a99b709038f3"} Oct 30 09:23:44.125952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.125914 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:23:44.127686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.127657 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:44.127843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.127720 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:44.427989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.427945 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4748a6d-905a-4fa3-96e5-da9b5c834ef0" path="/var/lib/kubelet/pods/e4748a6d-905a-4fa3-96e5-da9b5c834ef0/volumes" Oct 30 09:23:44.686791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.686682 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:44.990467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:44.990369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:45.128681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.128644 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:45.128992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.128957 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:45.236667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.236631 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:23:45.243617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.243556 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.256866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.256832 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:23:45.259748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.259727 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-356553-openshift-default-dockercfg-dmnvm\"" Oct 30 09:23:45.271930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.271887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.348886 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.348963 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349015 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349084 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349206 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349280 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcpp\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.349415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349391 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.351476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349506 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.351476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.349544 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457237 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457303 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457409 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457455 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457501 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457558 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457671 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457715 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcpp\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.457797 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.458434 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.458703 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.458727 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.458984 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.461239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.459491 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.511110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.496657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:45.511110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.498926 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.511110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.499617 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.511110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.510990 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcpp\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.526358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.526298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\") pod \"e2e-ipp-b-356553-openshift-default-766c847c47-482qh\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:45.572260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:45.571974 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:46.068035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.068006 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:23:46.068493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.068055 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:46.068493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.068115 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:46.068493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.068151 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:46.128201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.127748 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:46.128201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.127805 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:46.724168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:46.724131 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:47.028518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.028417 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerStarted","Data":"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb"} Oct 30 09:23:47.028518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.028483 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerStarted","Data":"26a938c5415aa2a24f9f35ed03f33fa4f8c18fa6df913211ea0509a397886601"} Oct 30 09:23:47.092180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.092105 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podStartSLOduration=2.092085935 podStartE2EDuration="2.092085935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:47.081589015 +0000 UTC m=+1790.805016160" watchObservedRunningTime="2026-10-30 09:23:47.092085935 +0000 UTC m=+1790.815513070" Oct 30 09:23:47.127241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.126696 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:47.127241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.126762 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:47.580999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.580965 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:47.586619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:47.586588 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:48.037580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:48.037555 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:48.038868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:48.038840 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:23:48.132246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:48.129685 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:48.132246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:48.129750 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:48.367375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:48.367275 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:49.041986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.041950 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl"] Oct 30 09:23:49.049688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.049655 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef722d31-nbjsl"] Oct 30 09:23:49.127048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.126609 4920 patch_prober.go:29] interesting pod/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 09:23:49.127048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.126686 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 09:23:49.428014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.427922 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:49.539387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:49.539351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:50.132388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:50.132343 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:23:50.132880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:50.132862 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:23:50.133792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:50.133769 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:23:50.215158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:50.215119 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:50.401068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:50.400976 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f57307-f893-4963-bffd-f23aa71b71c3" path="/var/lib/kubelet/pods/f5f57307-f893-4963-bffd-f23aa71b71c3/volumes" Oct 30 09:23:51.037111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:51.037077 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8"] Oct 30 09:23:51.049081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:51.049049 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45af605c-mpvt8"] Oct 30 09:23:51.154527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:51.154497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:51.355549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:51.355465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:51.948045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:51.948020 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:23:52.402651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:52.402614 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1390ba2-5973-4111-bd72-c0da68514966" path="/var/lib/kubelet/pods/b1390ba2-5973-4111-bd72-c0da68514966/volumes" Oct 30 09:23:52.582693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:52.582655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:52.883413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:52.883377 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:54.521422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:54.521388 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:54.864680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:54.864646 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:56.736577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:56.736537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:56.834525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:56.834468 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:58.746478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:58.746035 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:58.869111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:58.869071 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:23:59.671234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:59.671188 4920 scope.go:122] "RemoveContainer" containerID="a9bdc2227b756d0c283a8e03cae3cf53ac7311a52b8e6c877b9b9f5a4f319be9" Oct 30 09:23:59.682236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:23:59.682201 4920 scope.go:122] "RemoveContainer" containerID="55f6e77ee7c3c5b382031054fbbb4d11d745af29557459d6fd48dc81fea1b751" Oct 30 09:24:00.380048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.380013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:00.485170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.485136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:00.614328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.614289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:00.877254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.869957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:00.994276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.983029 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:24:00.994276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:00.983317 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" containerID="cri-o://dd0302aae6644f036606239568757ab33bde86992d40d6eefff68bd475a1265d" gracePeriod=30 Oct 30 09:24:01.068579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:01.068543 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w"] Oct 30 09:24:01.136293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:01.134294 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e874d9c8-w2p2w"] Oct 30 09:24:01.213279 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:01.213156 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:24:01.721241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:01.718933 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:02.152296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.152255 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:24:02.159922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.159894 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.180092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.180060 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:24:02.185855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.185828 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4fc2e2-openshift-default-dockercfg-wbjcd\"" Oct 30 09:24:02.192795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.192765 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.192934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.192809 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.192975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.192957 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.192983 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.193093 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jz8d\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.193142 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.193168 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.193184 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.193361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.193343 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343495 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343700 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343741 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343843 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343876 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343918 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jz8d\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.343962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343953 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.344048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.343991 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.344048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.344020 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.360017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.344609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.360017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.344897 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.377339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.373645 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.552668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.376893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.552668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.378069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:02.552668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.473725 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.552668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.548869 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.603238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.598910 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944b6d05-12c6-43ef-8dae-300bd0db8e08" path="/var/lib/kubelet/pods/944b6d05-12c6-43ef-8dae-300bd0db8e08/volumes" Oct 30 09:24:02.682239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.672923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.682239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.675020 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.682239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.675846 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jz8d\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\") pod \"e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:02.688239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.686135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:02.785345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:02.783296 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:03.094498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.094410 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:24:03.149587 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:24:03.149553 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50341379_878a_4890_b526_1ac48747bde4.slice/crio-bbe9484040b8b175cc9c21e75e4696497adc5ce8d7f6761f102f83b4cf60656c WatchSource:0}: Error finding container bbe9484040b8b175cc9c21e75e4696497adc5ce8d7f6761f102f83b4cf60656c: Status 404 returned error can't find the container with id bbe9484040b8b175cc9c21e75e4696497adc5ce8d7f6761f102f83b4cf60656c Oct 30 09:24:03.150472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.150295 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:24:03.150566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.150504 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:24:03.152608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.152571 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:03.152703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.152684 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:03.152761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.152725 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:03.656255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.656182 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerStarted","Data":"b3a23f420c9ca12076115b427640137eb699867bd9e9ba24ceb2080383c07118"} Oct 30 09:24:03.656462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.656267 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerStarted","Data":"bbe9484040b8b175cc9c21e75e4696497adc5ce8d7f6761f102f83b4cf60656c"} Oct 30 09:24:03.714268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.714186 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podStartSLOduration=1.714165881 podStartE2EDuration="1.714165881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:03.696179613 +0000 UTC m=+1807.419606761" watchObservedRunningTime="2026-10-30 09:24:03.714165881 +0000 UTC m=+1807.437593025" Oct 30 09:24:03.783554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.783520 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:03.988141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:03.988061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:04.790196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:04.790165 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:05.030728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:05.030694 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp"] Oct 30 09:24:05.037116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:05.037085 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3741f808-8nrxp"] Oct 30 09:24:05.676941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:05.676910 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:05.677871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:05.677848 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:24:06.032986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.032900 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh"] Oct 30 09:24:06.042285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.042030 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c935a589-tjhjh"] Oct 30 09:24:06.060992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.060931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:06.172603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.172562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:06.557682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.557650 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123aa87d-d965-43db-9189-4fd0308b3dcc" path="/var/lib/kubelet/pods/123aa87d-d965-43db-9189-4fd0308b3dcc/volumes" Oct 30 09:24:06.558539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.558511 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9d39d0-8483-490b-a713-2d6d27389a4e" path="/var/lib/kubelet/pods/1f9d39d0-8483-490b-a713-2d6d27389a4e/volumes" Oct 30 09:24:06.689363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.689334 4920 generic.go:356] "Generic (PLEG): container finished" podID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerID="dd0302aae6644f036606239568757ab33bde86992d40d6eefff68bd475a1265d" exitCode=0 Oct 30 09:24:06.689538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.689428 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerDied","Data":"dd0302aae6644f036606239568757ab33bde86992d40d6eefff68bd475a1265d"} Oct 30 09:24:06.689538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.689523 4920 scope.go:122] "RemoveContainer" containerID="4a7776718e62660ed5cbd6010a9d57c37b3e55d80779ed0bf8cc38e3e55a1bd8" Oct 30 09:24:06.722449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.722409 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:24:06.722636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.722480 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:24:06.772546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.772511 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:24:06.876065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.875972 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876318 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:06.876454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876377 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876469 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876585 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876654 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876703 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876795 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876848 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.876876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876654 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:06.876978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.876955 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\") pod \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\" (UID: \"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148\") " Oct 30 09:24:06.877592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.877419 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:06.877741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.877717 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.877901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.877885 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.878001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.877988 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.878084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.877833 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:06.878177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.878062 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:06.879455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.879424 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:06.879568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.879507 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:06.880285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.880257 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:06.880881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.880854 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c" pod "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" (UID: "5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148"). InnerVolumeSpecName "kube-api-access-s4g5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:06.980121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980083 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.980121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980116 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.980121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980126 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.980476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980137 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4g5c\" (UniqueName: \"kubernetes.io/projected/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-kube-api-access-s4g5c\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.980476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980146 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:06.980476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:06.980154 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:07.035155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.035105 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx"] Oct 30 09:24:07.039770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.039738 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20caa23c-q5ntx"] Oct 30 09:24:07.700841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.700798 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" event={"ID":"5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148","Type":"ContainerDied","Data":"6254d3c4c8c97b10cc5bdba9150b027f883f0a36a22b103e4aae9d76effad092"} Oct 30 09:24:07.700841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.700827 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb" Oct 30 09:24:07.701111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.700876 4920 scope.go:122] "RemoveContainer" containerID="dd0302aae6644f036606239568757ab33bde86992d40d6eefff68bd475a1265d" Oct 30 09:24:07.779081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.779050 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:24:07.786599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.786567 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8eada0cf-openshift-default-685777678f-9x9hb"] Oct 30 09:24:07.792645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.792615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:07.889437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:07.889402 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:08.353329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.352346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:08.427345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.426776 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c9bbd6-4104-4a17-ae06-7227131fdff4" path="/var/lib/kubelet/pods/11c9bbd6-4104-4a17-ae06-7227131fdff4/volumes" Oct 30 09:24:08.428627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.428074 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" path="/var/lib/kubelet/pods/5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148/volumes" Oct 30 09:24:08.716348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.716319 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerID="44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026" exitCode=0 Oct 30 09:24:08.716524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.716393 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerDied","Data":"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026"} Oct 30 09:24:08.716907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.716885 4920 scope.go:122] "RemoveContainer" containerID="44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026" Oct 30 09:24:08.718690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.718650 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:08.719089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.718745 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:08.719089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:08.718821 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:09.759780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:09.758500 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerStarted","Data":"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730"} Oct 30 09:24:09.849854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:09.849808 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:10.046719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.046637 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f"] Oct 30 09:24:10.054001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.053969 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa4640-7zf7f"] Oct 30 09:24:10.153526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.153491 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:24:10.154613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.154584 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:10.154756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.154640 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:10.423132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:10.423036 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e04fa0-c4f4-4e25-8a83-c3172072bc9b" path="/var/lib/kubelet/pods/d7e04fa0-c4f4-4e25-8a83-c3172072bc9b/volumes" Oct 30 09:24:11.154584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:11.154547 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:11.155092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:11.154622 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:11.330239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:11.329697 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:12.153798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:12.153762 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:12.153989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:12.153822 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:12.849964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:12.849931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:12.976071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:12.976023 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:13.073273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.073236 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:13.154322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.154136 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:13.154322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.154197 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:13.357260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.357189 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:13.672102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.672017 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:13.784313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:13.783135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:14.158105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:14.158064 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:14.158815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:14.158772 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:14.953467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:14.953433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:15.154028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:15.153983 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 09:24:15.154206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:15.154065 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 09:24:16.158069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:16.158032 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:24:16.158487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:16.158473 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:24:16.159123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:16.159102 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:24:16.590897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:16.590861 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:18.649196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:18.649164 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:20.033296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:20.033261 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v"] Oct 30 09:24:20.043386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:20.043358 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129e40f5-4qm5v"] Oct 30 09:24:20.405835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:20.405788 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d4dd57-2eb7-45f1-b53d-ff9d2619b164" path="/var/lib/kubelet/pods/70d4dd57-2eb7-45f1-b53d-ff9d2619b164/volumes" Oct 30 09:24:20.564379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:20.564342 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:24:20.592556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:20.592516 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:22.724898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:22.724863 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:24.571067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:24.571025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:26.320368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:26.320328 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:26.460238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:26.455637 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:26.567242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:26.566259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:26.666336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:26.666247 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:27.323068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:27.323026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:28.032940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.032901 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4"] Oct 30 09:24:28.044742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.044712 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd08f3e8-zhhk4"] Oct 30 09:24:28.401724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.401676 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc28ea1-392d-4404-afe1-5158ad5c8f5d" path="/var/lib/kubelet/pods/dfc28ea1-392d-4404-afe1-5158ad5c8f5d/volumes" Oct 30 09:24:28.536975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.536944 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:28.579195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.579160 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh"] Oct 30 09:24:28.579947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.579927 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.580057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.579949 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.580057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.579986 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.580057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.579994 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.580208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.580182 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.580208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.580201 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6b2f06-9e0e-4f39-ba1d-f3e1c7a6f148" containerName="istio-proxy" Oct 30 09:24:28.585314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.585292 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.620496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.613205 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh"] Oct 30 09:24:28.639911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.629044 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:24:28.640044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.629159 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:24:28.640126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.629185 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:24:28.641678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.641648 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:24:28.749318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.749212 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.749467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.749371 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jt4\" (UniqueName: \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.749467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.749399 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.851119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.851085 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.851315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.851201 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jt4\" (UniqueName: \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.851315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.851270 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.851921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.851899 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.853511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.853482 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.869963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.869938 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jt4\" (UniqueName: \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\") pod \"maas-api-revoke-keys-e2e-api-a-58fadf-chcqh\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:28.952281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:28.952241 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:29.266740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:29.266414 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh"] Oct 30 09:24:29.268572 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:24:29.268545 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa31827a_a37d_4515_a677_66b6f8f0a28a.slice/crio-e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10 WatchSource:0}: Error finding container e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10: Status 404 returned error can't find the container with id e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10 Oct 30 09:24:30.073153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:30.073115 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" event={"ID":"aa31827a-a37d-4515-a677-66b6f8f0a28a","Type":"ContainerStarted","Data":"95e6c37b30cc6d7a5788c379ff0c2896c553c46d300e302297b806ae90092134"} Oct 30 09:24:30.073632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:30.073170 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" event={"ID":"aa31827a-a37d-4515-a677-66b6f8f0a28a","Type":"ContainerStarted","Data":"e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10"} Oct 30 09:24:30.114283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:30.114229 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" podStartSLOduration=2.114200387 podStartE2EDuration="2.114200387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:30.112680834 +0000 UTC m=+1833.836107977" watchObservedRunningTime="2026-10-30 09:24:30.114200387 +0000 UTC m=+1833.837627530" Oct 30 09:24:30.319876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:30.319839 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:31.083807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:31.083773 4920 generic.go:356] "Generic (PLEG): container finished" podID="aa31827a-a37d-4515-a677-66b6f8f0a28a" containerID="95e6c37b30cc6d7a5788c379ff0c2896c553c46d300e302297b806ae90092134" exitCode=0 Oct 30 09:24:31.084274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:31.083853 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" event={"ID":"aa31827a-a37d-4515-a677-66b6f8f0a28a","Type":"ContainerDied","Data":"95e6c37b30cc6d7a5788c379ff0c2896c553c46d300e302297b806ae90092134"} Oct 30 09:24:31.140836 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:31.140786 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:24:32.275922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.275895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:32.294618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.294596 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:32.309849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.309818 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\") pod \"aa31827a-a37d-4515-a677-66b6f8f0a28a\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " Oct 30 09:24:32.310010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.309919 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\" (UniqueName: \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\") pod \"aa31827a-a37d-4515-a677-66b6f8f0a28a\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " Oct 30 09:24:32.310164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.310142 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca" pod "aa31827a-a37d-4515-a677-66b6f8f0a28a" (UID: "aa31827a-a37d-4515-a677-66b6f8f0a28a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:32.310302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.310282 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\") pod \"aa31827a-a37d-4515-a677-66b6f8f0a28a\" (UID: \"aa31827a-a37d-4515-a677-66b6f8f0a28a\") " Oct 30 09:24:32.311034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.311006 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:32.312627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.312594 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca" pod "aa31827a-a37d-4515-a677-66b6f8f0a28a" (UID: "aa31827a-a37d-4515-a677-66b6f8f0a28a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:24:32.312746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.312723 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4" pod "aa31827a-a37d-4515-a677-66b6f8f0a28a" (UID: "aa31827a-a37d-4515-a677-66b6f8f0a28a"). InnerVolumeSpecName "kube-api-access-p5jt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:32.359003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.358894 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:24:32.360182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.360154 4920 cpu_manager.go:414] "containerMap: removing container" podUID="aa31827a-a37d-4515-a677-66b6f8f0a28a" containerName="revoke-keys" Oct 30 09:24:32.360182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.360179 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa31827a-a37d-4515-a677-66b6f8f0a28a" containerName="revoke-keys" Oct 30 09:24:32.360439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.360375 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa31827a-a37d-4515-a677-66b6f8f0a28a" containerName="revoke-keys" Oct 30 09:24:32.364288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.364263 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.404089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.399998 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:24:32.404089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.401448 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9371b7-openshift-default-dockercfg-fccbl\"" Oct 30 09:24:32.419734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.418918 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa31827a-a37d-4515-a677-66b6f8f0a28a-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:32.419734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.418953 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5jt4\" (UniqueName: \"kubernetes.io/projected/aa31827a-a37d-4515-a677-66b6f8f0a28a-kube-api-access-p5jt4\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:32.459285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.459023 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.547649 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.547796 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.547892 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549286 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549635 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549668 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549743 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549771 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.551243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.549806 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxqq\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.651958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.651863 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.651958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.651913 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.651965 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.651986 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652018 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxqq\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652124 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652169 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652194 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652368 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652376 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652605 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652730 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652778 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.652842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.652805 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.654748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.654722 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.654900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.654858 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.671707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.671676 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.671947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.671927 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxqq\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\") pod \"e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:32.721721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:32.721684 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:33.080724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.079647 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:24:33.083995 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:24:33.083669 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4c20c8_2c3a_4129_851d_2776b81c95c9.slice/crio-29e8a5a46440b2569da6f3d69e1b8dc21c60396ed14331b42c35d0b44e12da7f WatchSource:0}: Error finding container 29e8a5a46440b2569da6f3d69e1b8dc21c60396ed14331b42c35d0b44e12da7f: Status 404 returned error can't find the container with id 29e8a5a46440b2569da6f3d69e1b8dc21c60396ed14331b42c35d0b44e12da7f Oct 30 09:24:33.088012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.087975 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:33.088157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.088065 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:33.088157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.088112 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:33.105003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.104975 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" Oct 30 09:24:33.105323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.105294 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh" event={"ID":"aa31827a-a37d-4515-a677-66b6f8f0a28a","Type":"ContainerDied","Data":"e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10"} Oct 30 09:24:33.105451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.105371 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2069815ebb35f3338e78b1084d3d341f96b98ef8c5d99243c05ecb767a2ad10" Oct 30 09:24:33.107155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:33.107080 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" event={"ID":"df4c20c8-2c3a-4129-851d-2776b81c95c9","Type":"ContainerStarted","Data":"29e8a5a46440b2569da6f3d69e1b8dc21c60396ed14331b42c35d0b44e12da7f"} Oct 30 09:24:34.029845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.029804 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:34.194147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.194104 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" event={"ID":"df4c20c8-2c3a-4129-851d-2776b81c95c9","Type":"ContainerStarted","Data":"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a"} Oct 30 09:24:34.279149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.279089 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podStartSLOduration=2.279072525 podStartE2EDuration="2.279072525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:34.277152495 +0000 UTC m=+1838.000579642" watchObservedRunningTime="2026-10-30 09:24:34.279072525 +0000 UTC m=+1838.002499671" Oct 30 09:24:34.727240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.723617 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:34.728593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.728560 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:34.728734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:34.728620 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:35.362954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:35.362921 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:35.479240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:35.478752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:35.736265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:35.735510 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:35.753119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:35.737565 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:36.724394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:36.724351 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:36.724905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:36.724419 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:36.771264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:36.771231 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:37.722139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:37.722101 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:37.722473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:37.722442 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:38.393287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:38.392688 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:38.755117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:38.746014 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:38.755117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:38.754337 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:39.282246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:39.281145 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:39.461668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:39.445309 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:39.723619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:39.723583 4920 patch_prober.go:29] interesting pod/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:24:39.723897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:39.723868 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 09:24:40.383679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:40.383633 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:40.733551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:40.733445 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:40.733989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:40.733881 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:40.734753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:40.734733 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:24:41.988924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:41.988891 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:43.674008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:43.673972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:45.505981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:45.505935 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:45.748464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:45.748424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:45.879746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:45.879707 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:45.979740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:45.979477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:47.753551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:47.753479 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:48.039626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.039590 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:48.039836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.039644 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:48.721923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.721889 4920 generic.go:356] "Generic (PLEG): container finished" podID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerID="c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb" exitCode=0 Oct 30 09:24:48.722135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.721971 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerDied","Data":"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb"} Oct 30 09:24:48.722516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.722499 4920 scope.go:122] "RemoveContainer" containerID="c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb" Oct 30 09:24:48.724186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.724155 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:48.724293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.724249 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:48.724293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:48.724285 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:49.628683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:49.628647 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:49.748904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:49.748868 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerStarted","Data":"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab"} Oct 30 09:24:49.768307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:49.768257 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:50.376490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.376447 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:24:50.376760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.376702 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" containerID="cri-o://1a62bb890e0882659995fd45e249af3280045a88c8122802e12fe0ce6b4b29b1" gracePeriod=30 Oct 30 09:24:50.388777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.388738 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:24:50.389025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.389002 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" containerID="cri-o://b67e544987bfb5044be6f58e704b4a2606a3c4316030a9be27dcc13efe23bcc3" gracePeriod=30 Oct 30 09:24:50.581053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.581011 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:24:50.581380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.581349 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:50.581505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:50.581408 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:51.431705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.431663 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:51.582654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.582607 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:51.582844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.582697 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:51.866301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.866268 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59"] Oct 30 09:24:51.870725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.870707 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:51.888717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.888112 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:24:51.888717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.888305 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:24:51.938347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.935946 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:24:51.938347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.936055 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:24:51.953342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.953303 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59"] Oct 30 09:24:51.955484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.955456 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:24:51.955716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.955686 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" containerID="cri-o://b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84" gracePeriod=30 Oct 30 09:24:51.964996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.964973 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:24:51.965334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.965303 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" containerID="cri-o://5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05" gracePeriod=30 Oct 30 09:24:51.971712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.971690 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:24:51.971911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.971889 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" containerID="cri-o://9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610" gracePeriod=30 Oct 30 09:24:51.977689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.977663 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:24:51.977903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.977879 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" containerID="cri-o://5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9" gracePeriod=30 Oct 30 09:24:51.991013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.990980 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:51.991160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.991112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:51.991278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:51.991187 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpfb\" (UniqueName: \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.092894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.092857 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.093082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.092920 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpfb\" (UniqueName: \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.093289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.093263 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.093528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.093509 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.095525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.095499 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.137465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.137348 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpfb\" (UniqueName: \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\") pod \"maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.185496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.185465 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:52.536555 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:24:52.536513 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f1a14d_0301_4331_9f7e_3dc41d66bef5.slice/crio-c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399 WatchSource:0}: Error finding container c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399: Status 404 returned error can't find the container with id c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399 Oct 30 09:24:52.554995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.554965 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59"] Oct 30 09:24:52.580830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.580801 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:52.580980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.580849 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:52.675820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.675787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:52.778145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.778113 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" event={"ID":"26f1a14d-0301-4331-9f7e-3dc41d66bef5","Type":"ContainerStarted","Data":"c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399"} Oct 30 09:24:52.856108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.856069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:52.933198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:52.933160 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:24:53.580618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:53.580581 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:53.581161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:53.581128 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:53.790023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:53.789987 4920 generic.go:356] "Generic (PLEG): container finished" podID="26f1a14d-0301-4331-9f7e-3dc41d66bef5" containerID="02f172203dd5cfb2ab09b327f802d191d5511ecff0743c093e40c211e484a960" exitCode=0 Oct 30 09:24:53.790256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:53.790096 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" event={"ID":"26f1a14d-0301-4331-9f7e-3dc41d66bef5","Type":"ContainerDied","Data":"02f172203dd5cfb2ab09b327f802d191d5511ecff0743c093e40c211e484a960"} Oct 30 09:24:54.219205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:54.219152 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:54.580570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:54.580533 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:54.580873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:54.580836 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:55.174379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.174355 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:55.266244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.265655 4920 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:24:55.266244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.265717 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:24:55.270247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.267638 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\") pod \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " Oct 30 09:24:55.270247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.267768 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\") pod \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " Oct 30 09:24:55.270247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.267993 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\" (UniqueName: \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\") pod \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\" (UID: \"26f1a14d-0301-4331-9f7e-3dc41d66bef5\") " Oct 30 09:24:55.270247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.269561 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca" pod "26f1a14d-0301-4331-9f7e-3dc41d66bef5" (UID: "26f1a14d-0301-4331-9f7e-3dc41d66bef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:55.276746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.276713 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb" pod "26f1a14d-0301-4331-9f7e-3dc41d66bef5" (UID: "26f1a14d-0301-4331-9f7e-3dc41d66bef5"). InnerVolumeSpecName "kube-api-access-5rpfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:55.276746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.276725 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca" pod "26f1a14d-0301-4331-9f7e-3dc41d66bef5" (UID: "26f1a14d-0301-4331-9f7e-3dc41d66bef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:24:55.370331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.370254 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:55.370562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.370549 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpfb\" (UniqueName: \"kubernetes.io/projected/26f1a14d-0301-4331-9f7e-3dc41d66bef5-kube-api-access-5rpfb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:55.370630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.370622 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f1a14d-0301-4331-9f7e-3dc41d66bef5-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:55.582104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.581684 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 09:24:55.582104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.581744 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 09:24:55.847233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.846794 4920 generic.go:356] "Generic (PLEG): container finished" podID="42896e60-9b13-48e6-92ab-cb4778354f68" containerID="1a62bb890e0882659995fd45e249af3280045a88c8122802e12fe0ce6b4b29b1" exitCode=0 Oct 30 09:24:55.847233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.846907 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" event={"ID":"42896e60-9b13-48e6-92ab-cb4778354f68","Type":"ContainerDied","Data":"1a62bb890e0882659995fd45e249af3280045a88c8122802e12fe0ce6b4b29b1"} Oct 30 09:24:55.850266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.849428 4920 generic.go:356] "Generic (PLEG): container finished" podID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerID="b67e544987bfb5044be6f58e704b4a2606a3c4316030a9be27dcc13efe23bcc3" exitCode=0 Oct 30 09:24:55.850266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.849680 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" event={"ID":"14134030-ed0b-46f1-bdbb-4d5a9edb4966","Type":"ContainerDied","Data":"b67e544987bfb5044be6f58e704b4a2606a3c4316030a9be27dcc13efe23bcc3"} Oct 30 09:24:55.853448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.852693 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" event={"ID":"26f1a14d-0301-4331-9f7e-3dc41d66bef5","Type":"ContainerDied","Data":"c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399"} Oct 30 09:24:55.853448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.852754 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15d0a46423eb429d5f945a9f00d28d2d4d9b564ac274a9f2c813258186e1399" Oct 30 09:24:55.853448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.853088 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59" Oct 30 09:24:55.872804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.872752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:55.887521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:55.887478 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:24:56.142051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142013 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142081 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142126 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142168 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142287 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142336 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142525 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142592 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.142746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.142642 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\") pod \"42896e60-9b13-48e6-92ab-cb4778354f68\" (UID: \"42896e60-9b13-48e6-92ab-cb4778354f68\") " Oct 30 09:24:56.144571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.144054 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.144571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.144069 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.144571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.144389 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:56.145369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.145318 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.145726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.145687 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.147463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.147436 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:56.149075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.149045 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.149485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.149447 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:56.151382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.151356 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9" pod "42896e60-9b13-48e6-92ab-cb4778354f68" (UID: "42896e60-9b13-48e6-92ab-cb4778354f68"). InnerVolumeSpecName "kube-api-access-tt2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:56.195900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.195872 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244038 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244139 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244237 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244272 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244314 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244361 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244418 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244450 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.244646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244597 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\") pod \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\" (UID: \"14134030-ed0b-46f1-bdbb-4d5a9edb4966\") " Oct 30 09:24:56.245181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244707 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.245181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244790 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.245181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.244943 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:56.245181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245035 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.245404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245373 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245396 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245406 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245415 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42896e60-9b13-48e6-92ab-cb4778354f68-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245424 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245433 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245445 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245463 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245480 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245493 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42896e60-9b13-48e6-92ab-cb4778354f68-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.245509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245504 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.248154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245518 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2t9\" (UniqueName: \"kubernetes.io/projected/42896e60-9b13-48e6-92ab-cb4778354f68-kube-api-access-tt2t9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.248154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.245531 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42896e60-9b13-48e6-92ab-cb4778354f68-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.248154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.246203 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.255186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.255158 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:56.255318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.255270 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "kube-api-access-khm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:56.256415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.256388 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:56.256741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.256708 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy" pod "14134030-ed0b-46f1-bdbb-4d5a9edb4966" (UID: "14134030-ed0b-46f1-bdbb-4d5a9edb4966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:56.346965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.346861 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.346965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.346898 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.346965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.346912 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.346965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.346924 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14134030-ed0b-46f1-bdbb-4d5a9edb4966-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.353323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.346997 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khm4r\" (UniqueName: \"kubernetes.io/projected/14134030-ed0b-46f1-bdbb-4d5a9edb4966-kube-api-access-khm4r\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:56.584689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.584662 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:24:56.585147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.585127 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:24:56.586049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.586029 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:24:56.864413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.864382 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" Oct 30 09:24:56.864615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.864407 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc" event={"ID":"42896e60-9b13-48e6-92ab-cb4778354f68","Type":"ContainerDied","Data":"55790b2b9258f03539387442f4b01eb0e1ed633a935f3f4da34e5fa8acecdbb3"} Oct 30 09:24:56.864615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.864486 4920 scope.go:122] "RemoveContainer" containerID="1a62bb890e0882659995fd45e249af3280045a88c8122802e12fe0ce6b4b29b1" Oct 30 09:24:56.866549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.866513 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" Oct 30 09:24:56.866876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.866548 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf" event={"ID":"14134030-ed0b-46f1-bdbb-4d5a9edb4966","Type":"ContainerDied","Data":"db8209cc69a5e5676ce9e7b61f0c2c15e8eb276808b08765dfc67f0f06e6a387"} Oct 30 09:24:56.878032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.877847 4920 scope.go:122] "RemoveContainer" containerID="b67e544987bfb5044be6f58e704b4a2606a3c4316030a9be27dcc13efe23bcc3" Oct 30 09:24:56.934362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.934321 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:24:56.938946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.938916 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-ds6kc"] Oct 30 09:24:56.959691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.959658 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:24:56.964024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:56.964000 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-gtnhf"] Oct 30 09:24:57.393002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.392763 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:24:57.475831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.475800 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.476101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.476079 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.476367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.476349 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.476549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.476535 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.476660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.476646 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.476896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.476852 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.478746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.477365 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:57.478746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.477652 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.478746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.478109 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.479032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.479016 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.480048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.480022 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.480550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.480533 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\") pod \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\" (UID: \"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2\") " Oct 30 09:24:57.480749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.479845 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.480831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.480471 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.481630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.481590 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.481774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.481758 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.481878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.481867 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.481964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.481952 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.482049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.482037 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.486959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.486929 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "kube-api-access-4577t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.488553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.488500 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.489279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.489257 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:57.490262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.490200 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:57.490350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.490274 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token" pod "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" (UID: "1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.584489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.582827 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.585483 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.585552 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.585589 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.585668 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.587336 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.587403 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.587520 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.587618 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.593105 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\") pod \"496e7447-55a8-416b-9a54-22ece43909c3\" (UID: \"496e7447-55a8-416b-9a54-22ece43909c3\") " Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.594574 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.594603 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.594631 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.594647 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4577t\" (UniqueName: \"kubernetes.io/projected/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2-kube-api-access-4577t\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.586605 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.586872 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.598933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.595115 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.608799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.608465 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.620501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.620417 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.668784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.663809 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:57.686372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.684281 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:24:57.688792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.688730 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.692328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.689435 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:57.692328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.690272 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52" pod "496e7447-55a8-416b-9a54-22ece43909c3" (UID: "496e7447-55a8-416b-9a54-22ece43909c3"). InnerVolumeSpecName "kube-api-access-csj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.694930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.694904 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:24:57.695439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695419 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695472 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695603 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695682 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695727 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695772 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695863 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695903 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.695994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.695974 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\") pod \"ff478156-3c2c-4a23-af6d-05771a5ccb17\" (UID: \"ff478156-3c2c-4a23-af6d-05771a5ccb17\") " Oct 30 09:24:57.696575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696557 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696577 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696592 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e7447-55a8-416b-9a54-22ece43909c3-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696604 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696617 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696629 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696641 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csj52\" (UniqueName: \"kubernetes.io/projected/496e7447-55a8-416b-9a54-22ece43909c3-kube-api-access-csj52\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696654 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e7447-55a8-416b-9a54-22ece43909c3-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.696669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696667 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e7447-55a8-416b-9a54-22ece43909c3-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.697094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.696937 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.697682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.697144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.699605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.698331 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.700097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.699916 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.700386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.700351 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:57.706407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.706368 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.709127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.709090 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.709364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.709331 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:57.709581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.709549 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6" pod "ff478156-3c2c-4a23-af6d-05771a5ccb17" (UID: "ff478156-3c2c-4a23-af6d-05771a5ccb17"). InnerVolumeSpecName "kube-api-access-s6mg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.797928 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798018 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798118 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798203 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798357 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798452 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798529 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798560 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.798586 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\") pod \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\" (UID: \"36ef2c48-92b4-4b4d-ace1-289e7b0ee771\") " Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799239 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799261 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799274 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799287 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff478156-3c2c-4a23-af6d-05771a5ccb17-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799301 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799313 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799326 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mg6\" (UniqueName: \"kubernetes.io/projected/ff478156-3c2c-4a23-af6d-05771a5ccb17-kube-api-access-s6mg6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799338 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799351 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff478156-3c2c-4a23-af6d-05771a5ccb17-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.799618 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.801303 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.801359 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.801399 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.809243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.801716 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:57.838832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.836980 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "kube-api-access-dw567". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.838832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.838744 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:24:57.842243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.842183 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:24:57.842985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.842955 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token" pod "36ef2c48-92b4-4b4d-ace1-289e7b0ee771" (UID: "36ef2c48-92b4-4b4d-ace1-289e7b0ee771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:57.879844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.879814 4920 generic.go:356] "Generic (PLEG): container finished" podID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerID="9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610" exitCode=0 Oct 30 09:24:57.880058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.880042 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" Oct 30 09:24:57.880181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.879983 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" event={"ID":"36ef2c48-92b4-4b4d-ace1-289e7b0ee771","Type":"ContainerDied","Data":"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610"} Oct 30 09:24:57.880600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.880582 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b" event={"ID":"36ef2c48-92b4-4b4d-ace1-289e7b0ee771","Type":"ContainerDied","Data":"6b7e5f2a6821003f9cb6abdff94327a8096d28cfae7e428b725cdaa874f1ca25"} Oct 30 09:24:57.880725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.880657 4920 scope.go:122] "RemoveContainer" containerID="9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610" Oct 30 09:24:57.894342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.893864 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" Oct 30 09:24:57.894342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.893885 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" event={"ID":"496e7447-55a8-416b-9a54-22ece43909c3","Type":"ContainerDied","Data":"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84"} Oct 30 09:24:57.894342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.893858 4920 generic.go:356] "Generic (PLEG): container finished" podID="496e7447-55a8-416b-9a54-22ece43909c3" containerID="b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84" exitCode=0 Oct 30 09:24:57.894342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.894186 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m" event={"ID":"496e7447-55a8-416b-9a54-22ece43909c3","Type":"ContainerDied","Data":"6603a281d99bf905fc864cb530e5ebbbbb73a19c89a5b96a3de162f95385e603"} Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922089 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922125 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922142 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922157 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922169 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922181 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw567\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-kube-api-access-dw567\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922195 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922209 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.928858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.922238 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ef2c48-92b4-4b4d-ace1-289e7b0ee771-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:24:57.935944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.930251 4920 generic.go:356] "Generic (PLEG): container finished" podID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerID="5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9" exitCode=0 Oct 30 09:24:57.935944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.930472 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" event={"ID":"ff478156-3c2c-4a23-af6d-05771a5ccb17","Type":"ContainerDied","Data":"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9"} Oct 30 09:24:57.935944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.930527 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" event={"ID":"ff478156-3c2c-4a23-af6d-05771a5ccb17","Type":"ContainerDied","Data":"c4136d18b874fdfdb182bb166e1b89c40d6148c6ee0785a0bcc45dfccad7608e"} Oct 30 09:24:57.935944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.930731 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr" Oct 30 09:24:57.944792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.941654 4920 generic.go:356] "Generic (PLEG): container finished" podID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerID="5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05" exitCode=0 Oct 30 09:24:57.944792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.941814 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerDied","Data":"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05"} Oct 30 09:24:57.944792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.941861 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" event={"ID":"1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2","Type":"ContainerDied","Data":"4042920146ab69825fe47ee0e36ada792cbe6e7e490d4d8a61867e6b15f9eb6e"} Oct 30 09:24:57.944792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.942025 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj" Oct 30 09:24:57.952759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.952302 4920 scope.go:122] "RemoveContainer" containerID="9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610" Oct 30 09:24:57.953133 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:57.952982 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610\": container with ID starting with 9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610 not found: ID does not exist" containerID="9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610" Oct 30 09:24:57.953133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.953026 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610"} err="failed to get container status \"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610\": rpc error: code = NotFound desc = could not find container \"9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610\": container with ID starting with 9e6548c592fa725bca04348097d6782bf4b6528c599736281cc6b9b6452e9610 not found: ID does not exist" Oct 30 09:24:57.953133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.953052 4920 scope.go:122] "RemoveContainer" containerID="b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84" Oct 30 09:24:57.968307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.968287 4920 scope.go:122] "RemoveContainer" containerID="b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84" Oct 30 09:24:57.968558 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:57.968539 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84\": container with ID starting with b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84 not found: ID does not exist" containerID="b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84" Oct 30 09:24:57.968648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.968566 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84"} err="failed to get container status \"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84\": rpc error: code = NotFound desc = could not find container \"b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84\": container with ID starting with b88f6c595ae41086753ce7c12642a5a67b01de1110952ecd93a1a3a7fa41ea84 not found: ID does not exist" Oct 30 09:24:57.968648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.968587 4920 scope.go:122] "RemoveContainer" containerID="5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9" Oct 30 09:24:57.980438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.980406 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:24:57.983730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.983704 4920 scope.go:122] "RemoveContainer" containerID="5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9" Oct 30 09:24:57.984058 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:57.984037 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9\": container with ID starting with 5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9 not found: ID does not exist" containerID="5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9" Oct 30 09:24:57.984153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.984068 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9"} err="failed to get container status \"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9\": rpc error: code = NotFound desc = could not find container \"5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9\": container with ID starting with 5969b74e1d274864f768228676ed83aac4891ab8f0684b8dbc5f1b24175a11a9 not found: ID does not exist" Oct 30 09:24:57.984153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.984086 4920 scope.go:122] "RemoveContainer" containerID="5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05" Oct 30 09:24:57.984651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.984630 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-wqc6m"] Oct 30 09:24:57.995645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:57.995622 4920 scope.go:122] "RemoveContainer" containerID="1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3" Oct 30 09:24:58.041912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.041886 4920 scope.go:122] "RemoveContainer" containerID="5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05" Oct 30 09:24:58.043287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.043247 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:24:58.044602 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:58.044568 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05\": container with ID starting with 5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05 not found: ID does not exist" containerID="5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05" Oct 30 09:24:58.044743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.044611 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05"} err="failed to get container status \"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05\": rpc error: code = NotFound desc = could not find container \"5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05\": container with ID starting with 5acdfada870ed38f3b191bf46b8f1bc31d67aaeba93492bca3182fb12b116b05 not found: ID does not exist" Oct 30 09:24:58.044743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.044637 4920 scope.go:122] "RemoveContainer" containerID="1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3" Oct 30 09:24:58.045023 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:24:58.045001 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3\": container with ID starting with 1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3 not found: ID does not exist" containerID="1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3" Oct 30 09:24:58.045059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.045034 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3"} err="failed to get container status \"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3\": rpc error: code = NotFound desc = could not find container \"1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3\": container with ID starting with 1bcb6298df64e156c16883dd006a4cf71ef65f4c30097b894c58a52565fa74a3 not found: ID does not exist" Oct 30 09:24:58.050283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.050261 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-q47mr"] Oct 30 09:24:58.083475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.083439 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:24:58.087710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.087685 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-7ld7b"] Oct 30 09:24:58.140013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.139978 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:24:58.144659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.144626 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-4vrpj"] Oct 30 09:24:58.422404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.422319 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" path="/var/lib/kubelet/pods/14134030-ed0b-46f1-bdbb-4d5a9edb4966/volumes" Oct 30 09:24:58.430268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.428871 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" path="/var/lib/kubelet/pods/1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2/volumes" Oct 30 09:24:58.430268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.429747 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" path="/var/lib/kubelet/pods/36ef2c48-92b4-4b4d-ace1-289e7b0ee771/volumes" Oct 30 09:24:58.430532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.430441 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" path="/var/lib/kubelet/pods/42896e60-9b13-48e6-92ab-cb4778354f68/volumes" Oct 30 09:24:58.431468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.431424 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496e7447-55a8-416b-9a54-22ece43909c3" path="/var/lib/kubelet/pods/496e7447-55a8-416b-9a54-22ece43909c3/volumes" Oct 30 09:24:58.432048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:58.432016 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" path="/var/lib/kubelet/pods/ff478156-3c2c-4a23-af6d-05771a5ccb17/volumes" Oct 30 09:24:59.189705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.189668 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:24:59.789338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.789297 4920 scope.go:122] "RemoveContainer" containerID="2152008e62bbc23e132c1be1fc0e195b4922a5697c28e79ec4f9395169516d6b" Oct 30 09:24:59.824431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.824402 4920 scope.go:122] "RemoveContainer" containerID="51ff463990be73fd9d81e2eaa01d6576598fdb757c87459348146b231d7ac456" Oct 30 09:24:59.841381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.841356 4920 scope.go:122] "RemoveContainer" containerID="18ba9606ddba5da57e790d73a809a71d891b61f2ad026b8a3458bdf69e9cdad9" Oct 30 09:24:59.852070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.852049 4920 scope.go:122] "RemoveContainer" containerID="4415b407a21780fcc264ec20883bae7acf0264bf8aea6a7d2cb836943f354cab" Oct 30 09:24:59.862126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.862102 4920 scope.go:122] "RemoveContainer" containerID="0397bc6dc72a7c7c2f8a4a72a89c8a2af72e6d36dfefc93c37489c8935ffa37a" Oct 30 09:24:59.872450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.872429 4920 scope.go:122] "RemoveContainer" containerID="06a8677e0c113212866f6cbaddc21aab0fdbae2e3eed59ce85958523a5aef948" Oct 30 09:24:59.885714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.885689 4920 scope.go:122] "RemoveContainer" containerID="5ffb1427ec4982bee5117568b0ca196e9fc9e81603584c4b0fac5c309bae59d1" Oct 30 09:24:59.896157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.896133 4920 scope.go:122] "RemoveContainer" containerID="f1d7952602cd6a6936eb4065eabfc938cd2d9294a15eb7ab18eb3c54efeb0037" Oct 30 09:24:59.932566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.932541 4920 scope.go:122] "RemoveContainer" containerID="17d952c5c4053362ad62e8fe33b023bcfe50e1b9cc2954f79740adca1e6da8a7" Oct 30 09:24:59.943116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.943096 4920 scope.go:122] "RemoveContainer" containerID="4a62509078d2d71365af5994d646cf64cf6fc1980f99ba6488ffa3c9baef902d" Oct 30 09:24:59.955489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.955469 4920 scope.go:122] "RemoveContainer" containerID="ef2a7d3f4b3dec81eb8b7aae3f33cddc54d509c2fb3d5b6048061812e80a73e1" Oct 30 09:24:59.965467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:24:59.965443 4920 scope.go:122] "RemoveContainer" containerID="8d0d79217a6c860f034e07fc218fffa8556cee8975184a66285bfd5e23f2023b" Oct 30 09:25:01.137364 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:01.137331 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:25:01.177240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:01.177015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:03.001410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:03.001371 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:05.054758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.054717 4920 generic.go:356] "Generic (PLEG): container finished" podID="50341379-878a-4890-b526-1ac48747bde4" containerID="b3a23f420c9ca12076115b427640137eb699867bd9e9ba24ceb2080383c07118" exitCode=0 Oct 30 09:25:05.055194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.054789 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerDied","Data":"b3a23f420c9ca12076115b427640137eb699867bd9e9ba24ceb2080383c07118"} Oct 30 09:25:05.055268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.055244 4920 scope.go:122] "RemoveContainer" containerID="b3a23f420c9ca12076115b427640137eb699867bd9e9ba24ceb2080383c07118" Oct 30 09:25:05.056992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.056965 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:05.057075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.057036 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:05.057075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.057069 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:05.119253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.117650 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:05.677507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:05.677468 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:25:06.064456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:06.064422 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerStarted","Data":"5cc8781d67f514033fbd0e89def3297a35f3282643d633603f2d0c69ce172a38"} Oct 30 09:25:06.652927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:06.652880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:06.785227 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:06.784584 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:25:06.789603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:06.789355 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 09:25:06.789603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:06.789417 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 09:25:07.428291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:07.424752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:07.784091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:07.784058 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 09:25:07.784269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:07.784126 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 09:25:08.126292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.125802 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:08.331980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.331943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:08.644608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.644548 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:08.665437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.665401 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:25:08.666157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666134 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666158 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666178 4920 cpu_manager.go:414] "containerMap: removing container" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666186 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666230 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666239 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666250 4920 cpu_manager.go:414] "containerMap: removing container" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" Oct 30 09:25:08.666277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666257 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666287 4920 cpu_manager.go:414] "containerMap: removing container" podUID="26f1a14d-0301-4331-9f7e-3dc41d66bef5" containerName="revoke-keys" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666294 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f1a14d-0301-4331-9f7e-3dc41d66bef5" containerName="revoke-keys" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666337 4920 cpu_manager.go:414] "containerMap: removing container" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666345 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666357 4920 cpu_manager.go:414] "containerMap: removing container" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666365 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666547 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666561 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="14134030-ed0b-46f1-bdbb-4d5a9edb4966" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666570 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="42896e60-9b13-48e6-92ab-cb4778354f68" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666585 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="496e7447-55a8-416b-9a54-22ece43909c3" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666595 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff478156-3c2c-4a23-af6d-05771a5ccb17" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666609 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ef2c48-92b4-4b4d-ace1-289e7b0ee771" containerName="istio-proxy" Oct 30 09:25:08.666675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.666620 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f1a14d-0301-4331-9f7e-3dc41d66bef5" containerName="revoke-keys" Oct 30 09:25:08.670873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.670850 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.672955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.672929 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:25:08.685868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.685422 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5491bf-openshift-default-dockercfg-n8pzj\"" Oct 30 09:25:08.750033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.750001 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:08.784756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.784108 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 09:25:08.784756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.784175 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862291 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862361 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862481 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcbwn\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862505 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862555 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862698 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862735 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.867240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.862795 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968477 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcbwn\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968541 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968594 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968629 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968736 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968771 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968834 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.968979 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.969299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.969028 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.970031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.969639 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.973240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.970238 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.973240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.970872 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.973240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.971355 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:08.973240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.972035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.010279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:08.997843 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.017238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.010764 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.017475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.011070 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.021236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.018239 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcbwn\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\") pod \"e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.191313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.191272 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:09.288942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.288901 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:09.438667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.438628 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb"] Oct 30 09:25:09.445623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.445586 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:09.445623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.445623 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:09.445860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.445814 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6ed2a7-3d04-461c-a1bd-f1e2068b2ba2" containerName="istio-proxy" Oct 30 09:25:09.451951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.451927 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb"] Oct 30 09:25:09.453293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.452205 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.467944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.467911 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:25:09.468082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.468010 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:25:09.468082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.468060 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:25:09.468250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.468114 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:25:09.585349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.585259 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkqv\" (UniqueName: \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.585566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.585546 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.585650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.585626 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.621242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.619668 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:09.648069 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:09.648040 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfebb963_2069_47e8_a15c_c6515b0af8a2.slice/crio-8d7790a0d4890face837029631b616f43dc31ea795110fea2e70d1da8506c19c WatchSource:0}: Error finding container 8d7790a0d4890face837029631b616f43dc31ea795110fea2e70d1da8506c19c: Status 404 returned error can't find the container with id 8d7790a0d4890face837029631b616f43dc31ea795110fea2e70d1da8506c19c Oct 30 09:25:09.650133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.650109 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:25:09.651330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.651297 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:09.651432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.651366 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:09.651432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.651401 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:25:09.687404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.687364 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.687605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.687439 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.687670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.687618 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkqv\" (UniqueName: \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.688136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.688113 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.694279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.691318 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.743797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.742012 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkqv\" (UniqueName: \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\") pod \"maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.785238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.773708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:09.785238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.773989 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:09.787495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.787047 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 09:25:09.787495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:09.787105 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 09:25:10.189512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.188962 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerStarted","Data":"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8"} Oct 30 09:25:10.189512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.189029 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerStarted","Data":"8d7790a0d4890face837029631b616f43dc31ea795110fea2e70d1da8506c19c"} Oct 30 09:25:10.455248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.448842 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:25:10.480633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.480602 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:25:10.480874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.480846 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:25:10.543673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.543645 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.551567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.547935 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podStartSLOduration=2.547918235 podStartE2EDuration="2.547918235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:10.507657174 +0000 UTC m=+1874.231084318" watchObservedRunningTime="2026-10-30 09:25:10.547918235 +0000 UTC m=+1874.271345376" Oct 30 09:25:10.553481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.553441 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.553726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.553705 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxvh\" (UniqueName: \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.559305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.559279 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-356553\"" Oct 30 09:25:10.560276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.560254 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-356553-dockercfg-svbkx\"" Oct 30 09:25:10.576730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.576285 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:25:10.576730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.576361 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:10.576730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.576388 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:25:10.586700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.585287 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:25:10.586700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.585460 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.698649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.698608 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkls\" (UniqueName: \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.699842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.699801 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.700341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.700318 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.700504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.700487 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxvh\" (UniqueName: \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.741298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.732051 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.765261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.747382 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxvh\" (UniqueName: \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\") pod \"payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:10.785260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.784808 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 09:25:10.785260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.784870 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 09:25:10.822240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.814680 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.822240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.814961 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkls\" (UniqueName: \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.822240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.816257 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.842365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.842240 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb"] Oct 30 09:25:10.843075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.843054 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkls\" (UniqueName: \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\") pod \"payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:10.846349 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:10.846321 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5038659_7424_4463_98ec_7b52cddefa4e.slice/crio-c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b WatchSource:0}: Error finding container c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b: Status 404 returned error can't find the container with id c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b Oct 30 09:25:10.879039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:10.879013 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:11.071074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.070889 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:11.085824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.085789 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:11.213276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.208549 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" event={"ID":"d5038659-7424-4463-98ec-7b52cddefa4e","Type":"ContainerStarted","Data":"c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b"} Oct 30 09:25:11.267964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.265843 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:25:11.295381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.295353 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:11.531748 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:11.531711 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1951fb_37b2_4dae_b970_5ee244546a14.slice/crio-5e392b6dce33506549296a6e0cfa0bf5b3bc08450ef0415e5521c99dc88172a3 WatchSource:0}: Error finding container 5e392b6dce33506549296a6e0cfa0bf5b3bc08450ef0415e5521c99dc88172a3: Status 404 returned error can't find the container with id 5e392b6dce33506549296a6e0cfa0bf5b3bc08450ef0415e5521c99dc88172a3 Oct 30 09:25:11.531748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.531734 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:25:11.634268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.633409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:11.919418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.916127 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:25:11.919418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.917884 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:25:11.919992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:11.919667 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:25:12.219257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.219137 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" event={"ID":"1c33b203-a7aa-482b-b211-2d597f4229e2","Type":"ContainerStarted","Data":"33baf38ed2ea1f27b58dace9dae6187ca8c66acd34d4b93d09a6cb6bd7f52699"} Oct 30 09:25:12.219520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.219494 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:12.219623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.219532 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" event={"ID":"1c33b203-a7aa-482b-b211-2d597f4229e2","Type":"ContainerStarted","Data":"457bdf783aa7b90ec81bad21cfc032088963a08d5efb5fa993ea4d1acae4486b"} Oct 30 09:25:12.221126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.221098 4920 generic.go:356] "Generic (PLEG): container finished" podID="d5038659-7424-4463-98ec-7b52cddefa4e" containerID="77e3a29e3cf2024f10ba58743a9573135cd257e044f5ccf7f5dede7924786de1" exitCode=0 Oct 30 09:25:12.221242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.221175 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" event={"ID":"d5038659-7424-4463-98ec-7b52cddefa4e","Type":"ContainerDied","Data":"77e3a29e3cf2024f10ba58743a9573135cd257e044f5ccf7f5dede7924786de1"} Oct 30 09:25:12.222975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.222952 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" event={"ID":"ae1951fb-37b2-4dae-b970-5ee244546a14","Type":"ContainerStarted","Data":"e20b02aab53b82ffc56d76b7d3ed1c67b1a19ddf9baaa76df26d7ae7da311439"} Oct 30 09:25:12.223082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.222997 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" event={"ID":"ae1951fb-37b2-4dae-b970-5ee244546a14","Type":"ContainerStarted","Data":"5e392b6dce33506549296a6e0cfa0bf5b3bc08450ef0415e5521c99dc88172a3"} Oct 30 09:25:12.223082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.223028 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:12.223386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.223370 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:12.224455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.224437 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:25:12.257092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.257029 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" podStartSLOduration=2.257012306 podStartE2EDuration="2.257012306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:12.256679444 +0000 UTC m=+1875.980106588" watchObservedRunningTime="2026-10-30 09:25:12.257012306 +0000 UTC m=+1875.980439449" Oct 30 09:25:12.302433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.302332 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" podStartSLOduration=2.302312557 podStartE2EDuration="2.302312557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:12.297035849 +0000 UTC m=+1876.020462997" watchObservedRunningTime="2026-10-30 09:25:12.302312557 +0000 UTC m=+1876.025739701" Oct 30 09:25:12.951324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:12.951288 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:13.064182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.064143 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:13.489951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.489925 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:13.600890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.600857 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\") pod \"d5038659-7424-4463-98ec-7b52cddefa4e\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " Oct 30 09:25:13.601054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.600984 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\" (UniqueName: \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\") pod \"d5038659-7424-4463-98ec-7b52cddefa4e\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " Oct 30 09:25:13.601143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.601111 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\") pod \"d5038659-7424-4463-98ec-7b52cddefa4e\" (UID: \"d5038659-7424-4463-98ec-7b52cddefa4e\") " Oct 30 09:25:13.601502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.601468 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca" pod "d5038659-7424-4463-98ec-7b52cddefa4e" (UID: "d5038659-7424-4463-98ec-7b52cddefa4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:13.601775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.601748 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:13.603183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.603144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca" pod "d5038659-7424-4463-98ec-7b52cddefa4e" (UID: "d5038659-7424-4463-98ec-7b52cddefa4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:13.603313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.603211 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv" pod "d5038659-7424-4463-98ec-7b52cddefa4e" (UID: "d5038659-7424-4463-98ec-7b52cddefa4e"). InnerVolumeSpecName "kube-api-access-trkqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:13.703689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.703654 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5038659-7424-4463-98ec-7b52cddefa4e-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:13.703689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:13.703683 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkqv\" (UniqueName: \"kubernetes.io/projected/d5038659-7424-4463-98ec-7b52cddefa4e-kube-api-access-trkqv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:14.250557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.250526 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" Oct 30 09:25:14.251013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.250560 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb" event={"ID":"d5038659-7424-4463-98ec-7b52cddefa4e","Type":"ContainerDied","Data":"c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b"} Oct 30 09:25:14.251013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.250615 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c397d65b76619c58f4bba3ab90ba46dc95fcd86fd3b21c6db517deeb6b65df2b" Oct 30 09:25:14.610861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.610817 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:14.717105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.717047 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:14.785332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.785289 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:25:14.785974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.785959 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d5038659-7424-4463-98ec-7b52cddefa4e" containerName="revoke-keys" Oct 30 09:25:14.786035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.785975 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5038659-7424-4463-98ec-7b52cddefa4e" containerName="revoke-keys" Oct 30 09:25:14.786126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.786116 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5038659-7424-4463-98ec-7b52cddefa4e" containerName="revoke-keys" Oct 30 09:25:14.790205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.790181 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:14.811618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.811587 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5491bf\"" Oct 30 09:25:14.811834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.811593 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5491bf-dockercfg-5rnz4\"" Oct 30 09:25:14.820250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.820192 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrs46\" (UniqueName: \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:14.820539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.820514 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:14.827193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.827159 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:25:14.941299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.938254 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:14.941299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.938387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrs46\" (UniqueName: \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:14.941299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:14.939858 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:15.065958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.065646 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrs46\" (UniqueName: \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\") pod \"payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:15.087112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.078989 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:25:15.100258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.093906 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:25:15.100258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.094034 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.117638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.117602 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:15.167460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.167425 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ln2\" (UniqueName: \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.167702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.167665 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.269108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.269074 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ln2\" (UniqueName: \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.269542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.269294 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.270206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.270182 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.288387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.288360 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ln2\" (UniqueName: \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\") pod \"payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.463076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.463045 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:15.467991 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:15.467958 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed3b183_6d23_4a6b_96e1_7fdafb77fb22.slice/crio-b36a383ff45c505c3b8fd315624a3284b493bf25cc2b8e205cf77182e649bd11 WatchSource:0}: Error finding container b36a383ff45c505c3b8fd315624a3284b493bf25cc2b8e205cf77182e649bd11: Status 404 returned error can't find the container with id b36a383ff45c505c3b8fd315624a3284b493bf25cc2b8e205cf77182e649bd11 Oct 30 09:25:15.470226 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.470189 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:25:15.482265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.482232 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:15.777286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:15.777160 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:25:15.777691 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:15.777663 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e13869_457b_4d01_a351_fc1fe72c920b.slice/crio-c9a5b7a5785c57dbed4bc0f09ba540c7135932a3240c05b7400d8e9cdeb3e81e WatchSource:0}: Error finding container c9a5b7a5785c57dbed4bc0f09ba540c7135932a3240c05b7400d8e9cdeb3e81e: Status 404 returned error can't find the container with id c9a5b7a5785c57dbed4bc0f09ba540c7135932a3240c05b7400d8e9cdeb3e81e Oct 30 09:25:16.272209 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.272178 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" event={"ID":"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22","Type":"ContainerStarted","Data":"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3"} Oct 30 09:25:16.272672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.272273 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:16.272672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.272300 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" event={"ID":"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22","Type":"ContainerStarted","Data":"b36a383ff45c505c3b8fd315624a3284b493bf25cc2b8e205cf77182e649bd11"} Oct 30 09:25:16.274051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.274021 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" event={"ID":"20e13869-457b-4d01-a351-fc1fe72c920b","Type":"ContainerStarted","Data":"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055"} Oct 30 09:25:16.274191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.274073 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" event={"ID":"20e13869-457b-4d01-a351-fc1fe72c920b","Type":"ContainerStarted","Data":"c9a5b7a5785c57dbed4bc0f09ba540c7135932a3240c05b7400d8e9cdeb3e81e"} Oct 30 09:25:16.274191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.274107 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:16.310969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.310892 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" podStartSLOduration=2.3108387710000002 podStartE2EDuration="2.310838771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:16.309370141 +0000 UTC m=+1880.032797285" watchObservedRunningTime="2026-10-30 09:25:16.310838771 +0000 UTC m=+1880.034265916" Oct 30 09:25:16.353585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:16.353524 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" podStartSLOduration=2.353503766 podStartE2EDuration="2.353503766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:16.347366133 +0000 UTC m=+1880.070793275" watchObservedRunningTime="2026-10-30 09:25:16.353503766 +0000 UTC m=+1880.076930911" Oct 30 09:25:17.554005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:17.553951 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:18.787355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.787317 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:25:18.787787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.787528 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" containerID="cri-o://0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd" gracePeriod=30 Oct 30 09:25:18.798257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.798197 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:25:18.798538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.798492 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" containerID="cri-o://c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414" gracePeriod=30 Oct 30 09:25:18.802887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.802857 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:25:18.803067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.803047 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" containerID="cri-o://705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2" gracePeriod=30 Oct 30 09:25:18.813790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.813757 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:25:18.814011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:18.813986 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" containerID="cri-o://c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c" gracePeriod=30 Oct 30 09:25:19.251301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:19.251195 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:20.177459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:20.177421 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:25:20.177955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:20.177486 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:25:21.283678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:21.283644 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:22.872942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:22.872885 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:23.253388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:23.253307 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:25:23.256282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:23.256258 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:25:23.757061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:23.757025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:24.155877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.155471 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:25:24.188756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.187800 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.239803 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.240182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.240278 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.240413 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.240590 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.240942 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241005 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241014 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241435 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241473 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241532 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.241758 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.242014 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.242527 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\") pod \"912d892e-fde1-47a5-bab8-6afe0468e0b4\" (UID: \"912d892e-fde1-47a5-bab8-6afe0468e0b4\") " Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.244066 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.244287 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245485 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245506 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245520 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912d892e-fde1-47a5-bab8-6afe0468e0b4-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245532 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245544 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245557 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.248813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.245569 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.251082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.251056 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "kube-api-access-zjhdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.251516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.251494 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy" pod "912d892e-fde1-47a5-bab8-6afe0468e0b4" (UID: "912d892e-fde1-47a5-bab8-6afe0468e0b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.259760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.259730 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:25:24.275277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.275165 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:25:24.346497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346461 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.346729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346517 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.346729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346580 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.346729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346617 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.346729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346696 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346747 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346757 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346774 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346859 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346932 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.346972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346938 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.346977 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347014 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347044 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347077 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347164 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347204 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347269 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.347338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347322 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\") pod \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\" (UID: \"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f\") " Oct 30 09:25:24.347852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347424 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\") pod \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\" (UID: \"8a4a02e3-58c3-4309-ad17-b4bbc670a35e\") " Oct 30 09:25:24.347959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347267 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.348019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347442 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.348019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347609 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.348019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.347793 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.348588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.348519 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:24.349867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.349421 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.349867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.349615 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.349867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.349701 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.349999 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350018 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350032 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350046 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350076 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350090 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912d892e-fde1-47a5-bab8-6afe0468e0b4-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350102 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350114 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350143 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350157 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350171 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhdf\" (UniqueName: \"kubernetes.io/projected/912d892e-fde1-47a5-bab8-6afe0468e0b4-kube-api-access-zjhdf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350185 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350199 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.350482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350322 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "kube-api-access-nb2j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.350873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.350696 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "kube-api-access-s4rbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.352940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.352915 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:24.354407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.354384 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:24.354564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.354548 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token" pod "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" (UID: "0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.354748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.354567 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.354748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.354702 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token" pod "8a4a02e3-58c3-4309-ad17-b4bbc670a35e" (UID: "8a4a02e3-58c3-4309-ad17-b4bbc670a35e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.384810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.384769 4920 generic.go:356] "Generic (PLEG): container finished" podID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerID="0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd" exitCode=0 Oct 30 09:25:24.385011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.384919 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerDied","Data":"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd"} Oct 30 09:25:24.385078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.385010 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" event={"ID":"0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f","Type":"ContainerDied","Data":"d2ba17acb92ac622f5ca8a2329ffd0d20f0fe86b3a9d9e84b8ef01a6ef6e02bd"} Oct 30 09:25:24.385078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.385042 4920 scope.go:122] "RemoveContainer" containerID="0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd" Oct 30 09:25:24.385257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.385239 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm" Oct 30 09:25:24.396002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.395965 4920 generic.go:356] "Generic (PLEG): container finished" podID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerID="c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c" exitCode=0 Oct 30 09:25:24.396924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.396902 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" Oct 30 09:25:24.400776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.399884 4920 generic.go:356] "Generic (PLEG): container finished" podID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerID="705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2" exitCode=0 Oct 30 09:25:24.400776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.400001 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" Oct 30 09:25:24.417318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.408658 4920 generic.go:356] "Generic (PLEG): container finished" podID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerID="c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414" exitCode=0 Oct 30 09:25:24.417318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.408796 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417347 4920 scope.go:122] "RemoveContainer" containerID="3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab" Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417410 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" event={"ID":"1f1cc4ba-1def-4d05-bf38-0caa04e46736","Type":"ContainerDied","Data":"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c"} Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417462 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc" event={"ID":"1f1cc4ba-1def-4d05-bf38-0caa04e46736","Type":"ContainerDied","Data":"9ead25c6fffeba7fd9b6fc07e413fcbee91d5a586513d3743306c2fd56614011"} Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417490 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" event={"ID":"8a4a02e3-58c3-4309-ad17-b4bbc670a35e","Type":"ContainerDied","Data":"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2"} Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417517 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq" event={"ID":"8a4a02e3-58c3-4309-ad17-b4bbc670a35e","Type":"ContainerDied","Data":"473318adfae39566d98877a83166b29435f45d64b59e9b121fc96a40ee26a7fa"} Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417540 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" event={"ID":"912d892e-fde1-47a5-bab8-6afe0468e0b4","Type":"ContainerDied","Data":"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414"} Oct 30 09:25:24.418199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.417566 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm" event={"ID":"912d892e-fde1-47a5-bab8-6afe0468e0b4","Type":"ContainerDied","Data":"1d7d90b7205343d02780fe68c9a113dc26562d1eaa3ca7fa33239f58859954f0"} Oct 30 09:25:24.434903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.434872 4920 scope.go:122] "RemoveContainer" containerID="0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd" Oct 30 09:25:24.435316 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:24.435288 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd\": container with ID starting with 0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd not found: ID does not exist" containerID="0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd" Oct 30 09:25:24.435425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.435328 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd"} err="failed to get container status \"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd\": rpc error: code = NotFound desc = could not find container \"0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd\": container with ID starting with 0f23465e2ab9d78502c178a8cb11dfd559e473c575398b0096c8868e070a4ddd not found: ID does not exist" Oct 30 09:25:24.435425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.435354 4920 scope.go:122] "RemoveContainer" containerID="3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab" Oct 30 09:25:24.435650 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:24.435627 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab\": container with ID starting with 3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab not found: ID does not exist" containerID="3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab" Oct 30 09:25:24.435718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.435659 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab"} err="failed to get container status \"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab\": rpc error: code = NotFound desc = could not find container \"3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab\": container with ID starting with 3096154c7e9622bed60baa2bb98b77671b4c0013d0a00186aee036cf01e7a9ab not found: ID does not exist" Oct 30 09:25:24.435718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.435679 4920 scope.go:122] "RemoveContainer" containerID="c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c" Oct 30 09:25:24.450332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.450305 4920 scope.go:122] "RemoveContainer" containerID="c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c" Oct 30 09:25:24.450854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.450820 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.450941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.450885 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.450984 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451084 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451187 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451293 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451341 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451378 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.451457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.451442 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\") pod \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\" (UID: \"1f1cc4ba-1def-4d05-bf38-0caa04e46736\") " Oct 30 09:25:24.452088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452066 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452095 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452110 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452125 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452137 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2j7\" (UniqueName: \"kubernetes.io/projected/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-kube-api-access-nb2j7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452151 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4rbb\" (UniqueName: \"kubernetes.io/projected/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f-kube-api-access-s4rbb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452164 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4a02e3-58c3-4309-ad17-b4bbc670a35e-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.452691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452637 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.452770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.452695 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.453507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.453484 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.453826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.453622 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.453945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.453782 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:24.454541 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:24.454397 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c\": container with ID starting with c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c not found: ID does not exist" containerID="c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c" Oct 30 09:25:24.454541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.454435 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c"} err="failed to get container status \"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c\": rpc error: code = NotFound desc = could not find container \"c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c\": container with ID starting with c941d7df992b1cd35500f793b138b658d7e0af47136a6b06d82c54d0144a878c not found: ID does not exist" Oct 30 09:25:24.454541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.454460 4920 scope.go:122] "RemoveContainer" containerID="705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2" Oct 30 09:25:24.455611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.455590 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:24.455759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.455595 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.455930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.455908 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "kube-api-access-zhbh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:24.457529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.457504 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy" pod "1f1cc4ba-1def-4d05-bf38-0caa04e46736" (UID: "1f1cc4ba-1def-4d05-bf38-0caa04e46736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:24.469985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.469959 4920 scope.go:122] "RemoveContainer" containerID="705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2" Oct 30 09:25:24.470383 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:24.470351 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2\": container with ID starting with 705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2 not found: ID does not exist" containerID="705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2" Oct 30 09:25:24.470472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.470393 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2"} err="failed to get container status \"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2\": rpc error: code = NotFound desc = could not find container \"705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2\": container with ID starting with 705fdbc339c99b31f6d856641c883ca334026c097ae27f12cfacdef65308e7a2 not found: ID does not exist" Oct 30 09:25:24.470472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.470417 4920 scope.go:122] "RemoveContainer" containerID="c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414" Oct 30 09:25:24.481247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.481203 4920 scope.go:122] "RemoveContainer" containerID="c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414" Oct 30 09:25:24.481530 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:24.481512 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414\": container with ID starting with c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414 not found: ID does not exist" containerID="c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414" Oct 30 09:25:24.481597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.481536 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414"} err="failed to get container status \"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414\": rpc error: code = NotFound desc = could not find container \"c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414\": container with ID starting with c773429f0369042ae368cc3dbcc8063da414edaf55b3a057b318a78f12be3414 not found: ID does not exist" Oct 30 09:25:24.495566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.495538 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:25:24.515888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.515855 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-kvrgm"] Oct 30 09:25:24.554507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554473 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhbh6\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-kube-api-access-zhbh6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554523 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554547 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554562 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554576 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554593 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554602 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1cc4ba-1def-4d05-bf38-0caa04e46736-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554620 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.554697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.554629 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1cc4ba-1def-4d05-bf38-0caa04e46736-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:24.573500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.573473 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:25:24.578391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.578364 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-6jtcm"] Oct 30 09:25:24.614902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.614867 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:25:24.618947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.618922 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-7blzq"] Oct 30 09:25:24.634127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.634085 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:24.772464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.772433 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:25:24.776646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:24.776620 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2n4zc"] Oct 30 09:25:25.956261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:25.956176 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:26.075781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.075742 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:25:26.076027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.075999 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" podUID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" containerName="payload-pre-processing" containerID="cri-o://e6853802f6cc324d6281f7e32504ed245ac8237109c101660c234843251fdafd" gracePeriod=30 Oct 30 09:25:26.125374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.125340 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:25:26.125643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.125614 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" containerID="cri-o://6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71" gracePeriod=30 Oct 30 09:25:26.128913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.128867 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:25:26.129351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.129323 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" containerID="cri-o://f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0" gracePeriod=30 Oct 30 09:25:26.139389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.138187 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:25:26.139389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.138448 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" containerID="cri-o://eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1" gracePeriod=30 Oct 30 09:25:26.140451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.140231 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:26.142827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.142792 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:25:26.143254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.143127 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" containerID="cri-o://f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659" gracePeriod=30 Oct 30 09:25:26.245493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.245461 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:26.654236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.648449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:26.654236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.651857 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" path="/var/lib/kubelet/pods/0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f/volumes" Oct 30 09:25:26.654236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.652631 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" path="/var/lib/kubelet/pods/1f1cc4ba-1def-4d05-bf38-0caa04e46736/volumes" Oct 30 09:25:26.657320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.657299 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" path="/var/lib/kubelet/pods/8a4a02e3-58c3-4309-ad17-b4bbc670a35e/volumes" Oct 30 09:25:26.658095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.658074 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" path="/var/lib/kubelet/pods/912d892e-fde1-47a5-bab8-6afe0468e0b4/volumes" Oct 30 09:25:26.658895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.658876 4920 generic.go:356] "Generic (PLEG): container finished" podID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" containerID="e6853802f6cc324d6281f7e32504ed245ac8237109c101660c234843251fdafd" exitCode=0 Oct 30 09:25:26.659109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.659093 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" event={"ID":"86208ffd-5a4f-4fe5-8171-bd9d3320f053","Type":"ContainerDied","Data":"e6853802f6cc324d6281f7e32504ed245ac8237109c101660c234843251fdafd"} Oct 30 09:25:26.662651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.662631 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:25:26.737129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.737074 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\") pod \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " Oct 30 09:25:26.737344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.737315 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\" (UniqueName: \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\") pod \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\" (UID: \"86208ffd-5a4f-4fe5-8171-bd9d3320f053\") " Oct 30 09:25:26.738058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.737500 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume" pod "86208ffd-5a4f-4fe5-8171-bd9d3320f053" (UID: "86208ffd-5a4f-4fe5-8171-bd9d3320f053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:26.738576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.738546 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86208ffd-5a4f-4fe5-8171-bd9d3320f053-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.742795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.742764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8" pod "86208ffd-5a4f-4fe5-8171-bd9d3320f053" (UID: "86208ffd-5a4f-4fe5-8171-bd9d3320f053"). InnerVolumeSpecName "kube-api-access-f7zs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:26.793749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.793717 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:26.799519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.799494 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:25:26.799654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.799564 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:25:26.839810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.839777 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7zs8\" (UniqueName: \"kubernetes.io/projected/86208ffd-5a4f-4fe5-8171-bd9d3320f053-kube-api-access-f7zs8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.854225 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.854182 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59"] Oct 30 09:25:26.858175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.858145 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8de308-kdj59"] Oct 30 09:25:26.892888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:26.892857 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:27.285135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.285106 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:25:27.285665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.285186 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:25:27.336161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.336129 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:27.669427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.669348 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" Oct 30 09:25:27.669427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.669370 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl" event={"ID":"86208ffd-5a4f-4fe5-8171-bd9d3320f053","Type":"ContainerDied","Data":"2fdc80329303454e512709f0a6960c91782de8a89a1fb8e7f1f39e76e2af3f88"} Oct 30 09:25:27.669642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.669448 4920 scope.go:122] "RemoveContainer" containerID="e6853802f6cc324d6281f7e32504ed245ac8237109c101660c234843251fdafd" Oct 30 09:25:27.723669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.723636 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:25:27.730046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:27.730018 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-bcbcc946c-hrwsl"] Oct 30 09:25:28.400237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.400177 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f1a14d-0301-4331-9f7e-3dc41d66bef5" path="/var/lib/kubelet/pods/26f1a14d-0301-4331-9f7e-3dc41d66bef5/volumes" Oct 30 09:25:28.400865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.400842 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" path="/var/lib/kubelet/pods/86208ffd-5a4f-4fe5-8171-bd9d3320f053/volumes" Oct 30 09:25:28.588099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.588065 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:28.858731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.858699 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5"] Oct 30 09:25:28.859658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859641 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:28.859707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859662 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:28.859748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859718 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" Oct 30 09:25:28.859748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859726 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" Oct 30 09:25:28.859806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859756 4920 cpu_manager.go:414] "containerMap: removing container" podUID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" containerName="payload-pre-processing" Oct 30 09:25:28.859806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859764 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" containerName="payload-pre-processing" Oct 30 09:25:28.859806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859778 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" Oct 30 09:25:28.859806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859785 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" Oct 30 09:25:28.859924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859806 4920 cpu_manager.go:414] "containerMap: removing container" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" Oct 30 09:25:28.859924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859814 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" Oct 30 09:25:28.859986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859946 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="86208ffd-5a4f-4fe5-8171-bd9d3320f053" containerName="payload-pre-processing" Oct 30 09:25:28.859986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859957 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4a02e3-58c3-4309-ad17-b4bbc670a35e" containerName="istio-proxy" Oct 30 09:25:28.859986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859972 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:28.859986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859982 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="912d892e-fde1-47a5-bab8-6afe0468e0b4" containerName="istio-proxy" Oct 30 09:25:28.860101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.859990 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:28.860101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.860006 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1cc4ba-1def-4d05-bf38-0caa04e46736" containerName="istio-proxy" Oct 30 09:25:28.865856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.865833 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:28.881601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.881435 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5"] Oct 30 09:25:28.886942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.886907 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:25:28.887124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.886980 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:25:28.893402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.893375 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:25:28.972121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.972086 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7t7m\" (UniqueName: \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:28.972331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.972204 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:28.972404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.972380 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:28.980968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:28.980949 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:25:29.073683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.073647 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7t7m\" (UniqueName: \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.073857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.073710 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.073857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.073774 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.074426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.074403 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.076080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.076061 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.092600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.092568 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7t7m\" (UniqueName: \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.183692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.183611 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:29.489298 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:29.489272 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0a387a_1824_4131_88c8_4a27437836a8.slice/crio-bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006 WatchSource:0}: Error finding container bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006: Status 404 returned error can't find the container with id bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006 Oct 30 09:25:29.491546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.491520 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5"] Oct 30 09:25:29.699268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:29.699189 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" event={"ID":"8f0a387a-1824-4131-88c8-4a27437836a8","Type":"ContainerStarted","Data":"bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006"} Oct 30 09:25:30.710553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:30.710514 4920 generic.go:356] "Generic (PLEG): container finished" podID="8f0a387a-1824-4131-88c8-4a27437836a8" containerID="3c9f2995fbf48255cbafee87ae048ee3bdba05c912213834d876f7643fd9231e" exitCode=0 Oct 30 09:25:30.710979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:30.710596 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" event={"ID":"8f0a387a-1824-4131-88c8-4a27437836a8","Type":"ContainerDied","Data":"3c9f2995fbf48255cbafee87ae048ee3bdba05c912213834d876f7643fd9231e"} Oct 30 09:25:30.847411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:30.847376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:31.140539 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.140442 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:25:31.488747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.488715 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:25:31.529303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.529275 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:25:31.532881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.532856 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:25:31.585835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.585810 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621713 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621761 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621830 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621868 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621911 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.621976 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622007 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622042 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622073 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622125 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622155 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622548 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.623807 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.627615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.627588 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.627810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.627792 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.622211 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634165 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634283 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634346 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634376 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634412 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634447 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634480 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634523 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634571 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634596 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634717 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634762 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634805 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634813 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634841 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634913 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\") pod \"8dd705ed-cf80-41b7-a90b-152da20bef54\" (UID: \"8dd705ed-cf80-41b7-a90b-152da20bef54\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634969 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.634977 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\") pod \"308ae586-e0fc-463c-afbc-3f3e00773bfd\" (UID: \"308ae586-e0fc-463c-afbc-3f3e00773bfd\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.635015 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\") pod \"c89df4fc-4e83-421d-881f-56e808d9f1f5\" (UID: \"c89df4fc-4e83-421d-881f-56e808d9f1f5\") " Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.635195 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.635299 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.635510 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.635525 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636121 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636148 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd705ed-cf80-41b7-a90b-152da20bef54-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636162 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636175 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636190 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636203 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636233 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636248 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89df4fc-4e83-421d-881f-56e808d9f1f5-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636261 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636274 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636288 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636300 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636759 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636805 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.636892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.636917 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.639462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.637528 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.648990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.648928 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.648990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.648928 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.649411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.649368 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.649737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.649678 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.649737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.649702 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5" pod "8dd705ed-cf80-41b7-a90b-152da20bef54" (UID: "8dd705ed-cf80-41b7-a90b-152da20bef54"). InnerVolumeSpecName "kube-api-access-db9m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.650067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.650043 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "kube-api-access-wfmf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.650315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.650275 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "kube-api-access-tzk27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.650957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.650931 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.651341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.651316 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.652336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.652300 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token" pod "308ae586-e0fc-463c-afbc-3f3e00773bfd" (UID: "308ae586-e0fc-463c-afbc-3f3e00773bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.654130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.654107 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token" pod "c89df4fc-4e83-421d-881f-56e808d9f1f5" (UID: "c89df4fc-4e83-421d-881f-56e808d9f1f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.735450 4920 generic.go:356] "Generic (PLEG): container finished" podID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerID="f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0" exitCode=0 Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.735550 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerDied","Data":"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0"} Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.735600 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" event={"ID":"c89df4fc-4e83-421d-881f-56e808d9f1f5","Type":"ContainerDied","Data":"defbdeecab7e3c493067e998f2393ed13a6e6c5b791e936a8758423510404e6b"} Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.735624 4920 scope.go:122] "RemoveContainer" containerID="f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.735735 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737421 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737471 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737522 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737578 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737606 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737630 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737662 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737685 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.737710 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\") pod \"fb406a82-75ea-427e-97b0-ac5c9805479c\" (UID: \"fb406a82-75ea-427e-97b0-ac5c9805479c\") " Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738148 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738237 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738747 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738770 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738785 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738799 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738812 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308ae586-e0fc-463c-afbc-3f3e00773bfd-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738826 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738841 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmf5\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-kube-api-access-wfmf5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738854 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738867 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9m5\" (UniqueName: \"kubernetes.io/projected/8dd705ed-cf80-41b7-a90b-152da20bef54-kube-api-access-db9m5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738880 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738893 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738905 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308ae586-e0fc-463c-afbc-3f3e00773bfd-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738919 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89df4fc-4e83-421d-881f-56e808d9f1f5-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738934 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb406a82-75ea-427e-97b0-ac5c9805479c-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738951 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738965 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd705ed-cf80-41b7-a90b-152da20bef54-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.738979 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzk27\" (UniqueName: \"kubernetes.io/projected/c89df4fc-4e83-421d-881f-56e808d9f1f5-kube-api-access-tzk27\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.739043 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.739105 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.739161 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.740185 4920 generic.go:356] "Generic (PLEG): container finished" podID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerID="eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1" exitCode=0 Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.740419 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" event={"ID":"8dd705ed-cf80-41b7-a90b-152da20bef54","Type":"ContainerDied","Data":"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1"} Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.740459 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" event={"ID":"8dd705ed-cf80-41b7-a90b-152da20bef54","Type":"ContainerDied","Data":"fdd9161720addb168ab9da49702a6bfd1f7ded414b907d929eb69e2a00ed6ecd"} Oct 30 09:25:31.741283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.740510 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t" Oct 30 09:25:31.752941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.746726 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:31.752941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.748589 4920 generic.go:356] "Generic (PLEG): container finished" podID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerID="f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659" exitCode=0 Oct 30 09:25:31.752941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.748769 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" event={"ID":"308ae586-e0fc-463c-afbc-3f3e00773bfd","Type":"ContainerDied","Data":"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659"} Oct 30 09:25:31.752941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.748819 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" event={"ID":"308ae586-e0fc-463c-afbc-3f3e00773bfd","Type":"ContainerDied","Data":"cf2bf37860ba69a2ecdb37f2b00a7310b0c8568f59ad883a406aaa6e92871503"} Oct 30 09:25:31.752941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.749170 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25" Oct 30 09:25:31.755335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.754237 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.756039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.755946 4920 generic.go:356] "Generic (PLEG): container finished" podID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerID="6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71" exitCode=0 Oct 30 09:25:31.756039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.755990 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" Oct 30 09:25:31.756308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.756209 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" event={"ID":"fb406a82-75ea-427e-97b0-ac5c9805479c","Type":"ContainerDied","Data":"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71"} Oct 30 09:25:31.756308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.756290 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn" event={"ID":"fb406a82-75ea-427e-97b0-ac5c9805479c","Type":"ContainerDied","Data":"2c29e40cad8276bb5eb2f310a60ff37cc715e519ae63bedec6fb4a59a57554c6"} Oct 30 09:25:31.759111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.759086 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:31.764697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.764657 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb" pod "fb406a82-75ea-427e-97b0-ac5c9805479c" (UID: "fb406a82-75ea-427e-97b0-ac5c9805479c"). InnerVolumeSpecName "kube-api-access-r6kdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:31.775939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.775914 4920 scope.go:122] "RemoveContainer" containerID="bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b" Oct 30 09:25:31.788063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.788043 4920 scope.go:122] "RemoveContainer" containerID="f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0" Oct 30 09:25:31.788459 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.788437 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0\": container with ID starting with f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0 not found: ID does not exist" containerID="f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0" Oct 30 09:25:31.788559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.788469 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0"} err="failed to get container status \"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0\": rpc error: code = NotFound desc = could not find container \"f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0\": container with ID starting with f7094019f131df9dfad681311c065adf22272c30c468d72e1cacd8e5edce97b0 not found: ID does not exist" Oct 30 09:25:31.788559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.788485 4920 scope.go:122] "RemoveContainer" containerID="bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b" Oct 30 09:25:31.788838 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.788798 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b\": container with ID starting with bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b not found: ID does not exist" containerID="bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b" Oct 30 09:25:31.788956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.788846 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b"} err="failed to get container status \"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b\": rpc error: code = NotFound desc = could not find container \"bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b\": container with ID starting with bf8a939b1b4972ac170dade499f16ec145d5d56b5f712927f8a2399a75babe4b not found: ID does not exist" Oct 30 09:25:31.788956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.788866 4920 scope.go:122] "RemoveContainer" containerID="eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1" Oct 30 09:25:31.822512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.817175 4920 scope.go:122] "RemoveContainer" containerID="eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1" Oct 30 09:25:31.822512 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.818074 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1\": container with ID starting with eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1 not found: ID does not exist" containerID="eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1" Oct 30 09:25:31.822512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.818110 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1"} err="failed to get container status \"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1\": rpc error: code = NotFound desc = could not find container \"eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1\": container with ID starting with eaeee100e3798caed8bda3fdafe0b6b7b8cc5cc6c353b564b66811726c6100e1 not found: ID does not exist" Oct 30 09:25:31.822512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.818135 4920 scope.go:122] "RemoveContainer" containerID="f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840275 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840304 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840323 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kdb\" (UniqueName: \"kubernetes.io/projected/fb406a82-75ea-427e-97b0-ac5c9805479c-kube-api-access-r6kdb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840337 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840351 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840365 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb406a82-75ea-427e-97b0-ac5c9805479c-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.840447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.840379 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb406a82-75ea-427e-97b0-ac5c9805479c-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:31.847943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.846465 4920 scope.go:122] "RemoveContainer" containerID="f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659" Oct 30 09:25:31.847943 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.846949 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659\": container with ID starting with f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659 not found: ID does not exist" containerID="f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659" Oct 30 09:25:31.847943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.846987 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659"} err="failed to get container status \"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659\": rpc error: code = NotFound desc = could not find container \"f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659\": container with ID starting with f6b5f9bad80713edd6887e7debe84ece89bb5ba6c8bc5f728bcc7848c3fcd659 not found: ID does not exist" Oct 30 09:25:31.847943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.847012 4920 scope.go:122] "RemoveContainer" containerID="6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71" Oct 30 09:25:31.849503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.849477 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:25:31.856185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.856146 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-g6n25"] Oct 30 09:25:31.865730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.865688 4920 scope.go:122] "RemoveContainer" containerID="6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71" Oct 30 09:25:31.866669 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:31.866175 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71\": container with ID starting with 6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71 not found: ID does not exist" containerID="6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71" Oct 30 09:25:31.866669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.866624 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71"} err="failed to get container status \"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71\": rpc error: code = NotFound desc = could not find container \"6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71\": container with ID starting with 6fcc74f44d9a2da689c804c717808bd5edf1245cbf5b75b931a228749656fa71 not found: ID does not exist" Oct 30 09:25:31.878274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.877980 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:25:31.882688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.882642 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kzf5s"] Oct 30 09:25:31.924321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.924289 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:25:31.928329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.928301 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtv4t"] Oct 30 09:25:31.958902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:31.958853 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:32.106147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.106103 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:25:32.110133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.110106 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-lzkkn"] Oct 30 09:25:32.144109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.144069 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\" (UniqueName: \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\") pod \"8f0a387a-1824-4131-88c8-4a27437836a8\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " Oct 30 09:25:32.144335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.144236 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\") pod \"8f0a387a-1824-4131-88c8-4a27437836a8\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " Oct 30 09:25:32.144335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.144297 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\") pod \"8f0a387a-1824-4131-88c8-4a27437836a8\" (UID: \"8f0a387a-1824-4131-88c8-4a27437836a8\") " Oct 30 09:25:32.144670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.144637 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca" pod "8f0a387a-1824-4131-88c8-4a27437836a8" (UID: "8f0a387a-1824-4131-88c8-4a27437836a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:32.144868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.144836 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:32.146402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.146380 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m" pod "8f0a387a-1824-4131-88c8-4a27437836a8" (UID: "8f0a387a-1824-4131-88c8-4a27437836a8"). InnerVolumeSpecName "kube-api-access-k7t7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:32.146728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.146712 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca" pod "8f0a387a-1824-4131-88c8-4a27437836a8" (UID: "8f0a387a-1824-4131-88c8-4a27437836a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:32.245994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.245950 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a387a-1824-4131-88c8-4a27437836a8-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:32.245994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.245993 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7t7m\" (UniqueName: \"kubernetes.io/projected/8f0a387a-1824-4131-88c8-4a27437836a8-kube-api-access-k7t7m\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:32.401382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.401294 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" path="/var/lib/kubelet/pods/308ae586-e0fc-463c-afbc-3f3e00773bfd/volumes" Oct 30 09:25:32.401966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.401939 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" path="/var/lib/kubelet/pods/8dd705ed-cf80-41b7-a90b-152da20bef54/volumes" Oct 30 09:25:32.402474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.402457 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" path="/var/lib/kubelet/pods/c89df4fc-4e83-421d-881f-56e808d9f1f5/volumes" Oct 30 09:25:32.403210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.403192 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" path="/var/lib/kubelet/pods/fb406a82-75ea-427e-97b0-ac5c9805479c/volumes" Oct 30 09:25:32.404655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.404627 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:32.812890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.812863 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" Oct 30 09:25:32.814110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.813480 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5" event={"ID":"8f0a387a-1824-4131-88c8-4a27437836a8","Type":"ContainerDied","Data":"bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006"} Oct 30 09:25:32.814513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:32.814487 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9932d892ae02440b47e25c10955e73940f0ac072b1d22c8f8b0936e9f0f006" Oct 30 09:25:33.206675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.206580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:33.386409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.386376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:33.423400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.423365 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh"] Oct 30 09:25:33.434518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.434483 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-58fadf-chcqh"] Oct 30 09:25:33.485952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.485874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:33.981863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:33.981831 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:34.402096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.402054 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa31827a-a37d-4515-a677-66b6f8f0a28a" path="/var/lib/kubelet/pods/aa31827a-a37d-4515-a677-66b6f8f0a28a/volumes" Oct 30 09:25:34.551949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.551916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:34.654632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.654538 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:34.891351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.891316 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:25:34.891653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.891622 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" containerID="cri-o://3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4" gracePeriod=30 Oct 30 09:25:34.910657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:34.910565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:35.607043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:35.607007 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:37.492138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:37.492099 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:25:37.569941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:37.569892 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:39.146345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146299 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:25:39.146927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146911 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:39.146927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146927 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146943 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8f0a387a-1824-4131-88c8-4a27437836a8" containerName="revoke-keys" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146948 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0a387a-1824-4131-88c8-4a27437836a8" containerName="revoke-keys" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146964 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146970 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6e2f02-fbc1-4c6f-8cbf-2bf84b823c1f" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146990 4920 cpu_manager.go:414] "containerMap: removing container" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.146995 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147002 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147007 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147014 4920 cpu_manager.go:414] "containerMap: removing container" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147019 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147046 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" Oct 30 09:25:39.147119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147050 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" Oct 30 09:25:39.147722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147138 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd705ed-cf80-41b7-a90b-152da20bef54" containerName="istio-proxy" Oct 30 09:25:39.147722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147147 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:39.147722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147153 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0a387a-1824-4131-88c8-4a27437836a8" containerName="revoke-keys" Oct 30 09:25:39.147722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147161 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb406a82-75ea-427e-97b0-ac5c9805479c" containerName="istio-proxy" Oct 30 09:25:39.147722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.147166 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="308ae586-e0fc-463c-afbc-3f3e00773bfd" containerName="istio-proxy" Oct 30 09:25:39.152487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.152465 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.174285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.167798 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:25:39.174285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.167904 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:25:39.174285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.172198 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:25:39.180023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.179995 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 09:25:39.180397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.180362 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hx9dk\"" Oct 30 09:25:39.180397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.180361 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5491bf-serving-cert\"" Oct 30 09:25:39.234615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.234583 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:39.261280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.261243 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.261463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.261399 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.261523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.261474 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.261656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.261632 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qsf\" (UniqueName: \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.349241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.348781 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:39.363511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.363468 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qsf\" (UniqueName: \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.363712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.363567 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.363712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.363676 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.363832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.363762 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.364648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.364616 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.366698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.366676 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.366807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.366676 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.385121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.385092 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qsf\" (UniqueName: \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\") pod \"maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.452000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.451912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:39.468063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.468025 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:39.779290 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:39.779261 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4dd8b8_0a42_461e_90f2_e122b193bc7c.slice/crio-5167d39f3d3922dae7946b688ed1e6ddf0c28701aff12a80a527775c58b22079 WatchSource:0}: Error finding container 5167d39f3d3922dae7946b688ed1e6ddf0c28701aff12a80a527775c58b22079: Status 404 returned error can't find the container with id 5167d39f3d3922dae7946b688ed1e6ddf0c28701aff12a80a527775c58b22079 Oct 30 09:25:39.780620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.780600 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:25:39.782039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.781223 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:25:39.876535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:39.876501 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" event={"ID":"8d4dd8b8-0a42-461e-90f2-e122b193bc7c","Type":"ContainerStarted","Data":"5167d39f3d3922dae7946b688ed1e6ddf0c28701aff12a80a527775c58b22079"} Oct 30 09:25:40.201642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.201608 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:25:40.400527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400185 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.400527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400403 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.400527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400451 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400554 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400598 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400653 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400690 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400756 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.400937 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\") pod \"b725a669-6638-4958-a8ca-e141e8999180\" (UID: \"b725a669-6638-4958-a8ca-e141e8999180\") " Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.402180 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.402571 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:40.420109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.402826 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.429854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.422312 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.429854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.426089 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.429854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.426305 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.432750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.430701 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.440824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.437016 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "kube-api-access-7k9dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.454419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.454381 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo" pod "b725a669-6638-4958-a8ca-e141e8999180" (UID: "b725a669-6638-4958-a8ca-e141e8999180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:40.507532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507492 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k9dh\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-kube-api-access-7k9dh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507532 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507546 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b725a669-6638-4958-a8ca-e141e8999180-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507559 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507573 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507587 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507599 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b725a669-6638-4958-a8ca-e141e8999180-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507612 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b725a669-6638-4958-a8ca-e141e8999180-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.507796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.507625 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b725a669-6638-4958-a8ca-e141e8999180-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.548876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.548839 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg"] Oct 30 09:25:40.549857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.549833 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" Oct 30 09:25:40.549977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.549960 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" Oct 30 09:25:40.550155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.550141 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89df4fc-4e83-421d-881f-56e808d9f1f5" containerName="istio-proxy" Oct 30 09:25:40.550273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.550160 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b725a669-6638-4958-a8ca-e141e8999180" containerName="istio-proxy" Oct 30 09:25:40.555366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.555342 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.572130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.572052 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:25:40.572130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.572102 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:25:40.573131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.573100 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg"] Oct 30 09:25:40.628687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.628618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:40.714519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.714429 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.714696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.714549 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvvh\" (UniqueName: \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.714696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.714616 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.740883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.740812 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:40.818249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.818058 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.818249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.818241 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvvh\" (UniqueName: \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.818509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.818354 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.837258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.819300 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.837258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.833935 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.852805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.852730 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvvh\" (UniqueName: \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.884883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.884575 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:40.958251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.956277 4920 generic.go:356] "Generic (PLEG): container finished" podID="b725a669-6638-4958-a8ca-e141e8999180" containerID="3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4" exitCode=0 Oct 30 09:25:40.958251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.956573 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" event={"ID":"b725a669-6638-4958-a8ca-e141e8999180","Type":"ContainerDied","Data":"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4"} Oct 30 09:25:40.958251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.957003 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" event={"ID":"b725a669-6638-4958-a8ca-e141e8999180","Type":"ContainerDied","Data":"7a3ddfbd62b19201e2c264607e3e61acedd820ea0407992f1f44a13bcdb0130d"} Oct 30 09:25:40.958251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.957051 4920 scope.go:122] "RemoveContainer" containerID="3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4" Oct 30 09:25:40.958251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.957265 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n" Oct 30 09:25:40.995490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.989187 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:25:40.995490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.989506 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" containerID="cri-o://bda354e62e5922babb734bf87526ffaa8cdd41ff466284f07734a99b709038f3" gracePeriod=30 Oct 30 09:25:40.995490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:40.994575 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.036239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.025497 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:25:41.036239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.034033 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-lzq6n"] Oct 30 09:25:41.060691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.059110 4920 scope.go:122] "RemoveContainer" containerID="3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4" Oct 30 09:25:41.060937 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:41.060906 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4\": container with ID starting with 3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4 not found: ID does not exist" containerID="3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4" Oct 30 09:25:41.061042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.060953 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4"} err="failed to get container status \"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4\": rpc error: code = NotFound desc = could not find container \"3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4\": container with ID starting with 3d78fb9e974f66c13cbacabfe2b848f30cb2c9a0c7be4e7e00c817fcb0b017a4 not found: ID does not exist" Oct 30 09:25:41.100868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.100799 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.214274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.208284 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:25:41.214734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.214583 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" podUID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" containerName="payload-processing" containerID="cri-o://1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3" gracePeriod=30 Oct 30 09:25:41.293260 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:41.293209 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443609f7_d69a_413b_88d7_7cd36d9cafae.slice/crio-6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0 WatchSource:0}: Error finding container 6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0: Status 404 returned error can't find the container with id 6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0 Oct 30 09:25:41.293906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.293878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.294312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.294289 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg"] Oct 30 09:25:41.404244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.402995 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.536655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.536602 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.562632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.562591 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz"] Oct 30 09:25:41.569444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.569416 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.587043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.587015 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz"] Oct 30 09:25:41.602549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.602443 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:25:41.694971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.694925 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:25:41.695258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.695206 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" podUID="a7151869-fb12-484e-8ce0-eb25159807a7" containerName="payload-processing" containerID="cri-o://18a904d52f587556899674dc6e14dc59c5116afbc4cb0b7501f75f21d70575a5" gracePeriod=30 Oct 30 09:25:41.747455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.747421 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\" (UniqueName: \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\") pod \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " Oct 30 09:25:41.747650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.747486 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\") pod \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\" (UID: \"b31fd1df-5a8a-4dcd-940d-17aae905fc95\") " Oct 30 09:25:41.747920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.747898 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.748011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.747992 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.748084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.748040 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwjf\" (UniqueName: \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.748638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.748613 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume" pod "b31fd1df-5a8a-4dcd-940d-17aae905fc95" (UID: "b31fd1df-5a8a-4dcd-940d-17aae905fc95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:41.751654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.751487 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.799111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.799036 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs" pod "b31fd1df-5a8a-4dcd-940d-17aae905fc95" (UID: "b31fd1df-5a8a-4dcd-940d-17aae905fc95"). InnerVolumeSpecName "kube-api-access-fwmvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:41.851013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.850974 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.851194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.851097 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.851194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.851141 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwjf\" (UniqueName: \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.851779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.851383 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmvs\" (UniqueName: \"kubernetes.io/projected/b31fd1df-5a8a-4dcd-940d-17aae905fc95-kube-api-access-fwmvs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.851779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.851406 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31fd1df-5a8a-4dcd-940d-17aae905fc95-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.853102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.853078 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.858593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.858550 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.866472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.866404 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:41.871884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.871824 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwjf\" (UniqueName: \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\") pod \"maas-api-revoke-keys-e2e-api-b-9371b7-mccfz\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:41.883568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.883505 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5"] Oct 30 09:25:41.889528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.889470 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f40a3-ffkh5"] Oct 30 09:25:41.928270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:41.928233 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:42.182621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.182556 4920 generic.go:356] "Generic (PLEG): container finished" podID="a7151869-fb12-484e-8ce0-eb25159807a7" containerID="18a904d52f587556899674dc6e14dc59c5116afbc4cb0b7501f75f21d70575a5" exitCode=0 Oct 30 09:25:42.183076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.183031 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" event={"ID":"a7151869-fb12-484e-8ce0-eb25159807a7","Type":"ContainerDied","Data":"18a904d52f587556899674dc6e14dc59c5116afbc4cb0b7501f75f21d70575a5"} Oct 30 09:25:42.186045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.186013 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" event={"ID":"443609f7-d69a-413b-88d7-7cd36d9cafae","Type":"ContainerStarted","Data":"d23b9418dcd6bde52f7745fb907ebc86c94f1dbb02c0c1fd21b22f215f0582f1"} Oct 30 09:25:42.186148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.186073 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" event={"ID":"443609f7-d69a-413b-88d7-7cd36d9cafae","Type":"ContainerStarted","Data":"6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0"} Oct 30 09:25:42.191130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.191099 4920 generic.go:356] "Generic (PLEG): container finished" podID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" containerID="1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3" exitCode=0 Oct 30 09:25:42.191468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.191440 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" event={"ID":"b31fd1df-5a8a-4dcd-940d-17aae905fc95","Type":"ContainerDied","Data":"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3"} Oct 30 09:25:42.191682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.191666 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" event={"ID":"b31fd1df-5a8a-4dcd-940d-17aae905fc95","Type":"ContainerDied","Data":"fba4a47901500fb6bd2ba18f100403e52831004b0c5b1dda11f266515b98287d"} Oct 30 09:25:42.191802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.191790 4920 scope.go:122] "RemoveContainer" containerID="1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3" Oct 30 09:25:42.192042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.192024 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw" Oct 30 09:25:42.242117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.242088 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:25:42.244174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.244120 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" podStartSLOduration=2.244102264 podStartE2EDuration="2.244102264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:42.234411285 +0000 UTC m=+1905.957838429" watchObservedRunningTime="2026-10-30 09:25:42.244102264 +0000 UTC m=+1905.967529408" Oct 30 09:25:42.244763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.244743 4920 scope.go:122] "RemoveContainer" containerID="1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3" Oct 30 09:25:42.245109 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:42.245087 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3\": container with ID starting with 1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3 not found: ID does not exist" containerID="1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3" Oct 30 09:25:42.245174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.245124 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3"} err="failed to get container status \"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3\": rpc error: code = NotFound desc = could not find container \"1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3\": container with ID starting with 1b68a225a38e1706bcaabd8cee71881ef15c2bc7ab25c36f00afed173bf71fa3 not found: ID does not exist" Oct 30 09:25:42.283722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.282714 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\") pod \"a7151869-fb12-484e-8ce0-eb25159807a7\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " Oct 30 09:25:42.283722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.282851 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\" (UniqueName: \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\") pod \"a7151869-fb12-484e-8ce0-eb25159807a7\" (UID: \"a7151869-fb12-484e-8ce0-eb25159807a7\") " Oct 30 09:25:42.284561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.284520 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume" pod "a7151869-fb12-484e-8ce0-eb25159807a7" (UID: "a7151869-fb12-484e-8ce0-eb25159807a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:42.289827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.289308 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:25:42.294436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.294342 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5" pod "a7151869-fb12-484e-8ce0-eb25159807a7" (UID: "a7151869-fb12-484e-8ce0-eb25159807a7"). InnerVolumeSpecName "kube-api-access-44gr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:42.309232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.309153 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f40a3-dcc45f766-4h6mw"] Oct 30 09:25:42.384715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.384633 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7151869-fb12-484e-8ce0-eb25159807a7-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:42.384715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.384670 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gr5\" (UniqueName: \"kubernetes.io/projected/a7151869-fb12-484e-8ce0-eb25159807a7-kube-api-access-44gr5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:42.409688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.403381 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0a387a-1824-4131-88c8-4a27437836a8" path="/var/lib/kubelet/pods/8f0a387a-1824-4131-88c8-4a27437836a8/volumes" Oct 30 09:25:42.409688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.403992 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" path="/var/lib/kubelet/pods/b31fd1df-5a8a-4dcd-940d-17aae905fc95/volumes" Oct 30 09:25:42.409688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.404655 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b725a669-6638-4958-a8ca-e141e8999180" path="/var/lib/kubelet/pods/b725a669-6638-4958-a8ca-e141e8999180/volumes" Oct 30 09:25:42.409688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.405303 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz"] Oct 30 09:25:42.519991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.519923 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:42.681185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.681098 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:42.738326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.738271 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb"] Oct 30 09:25:42.748749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:42.748714 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c44644-lzkmb"] Oct 30 09:25:43.189984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.189946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:43.203263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.202872 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" Oct 30 09:25:43.203263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.202894 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r" event={"ID":"a7151869-fb12-484e-8ce0-eb25159807a7","Type":"ContainerDied","Data":"4cab636b7901a2695dfed136a5fc872d0b9a829c44e3b710e0b356dcc16ce8d1"} Oct 30 09:25:43.203263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.202971 4920 scope.go:122] "RemoveContainer" containerID="18a904d52f587556899674dc6e14dc59c5116afbc4cb0b7501f75f21d70575a5" Oct 30 09:25:43.206066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.205603 4920 generic.go:356] "Generic (PLEG): container finished" podID="443609f7-d69a-413b-88d7-7cd36d9cafae" containerID="d23b9418dcd6bde52f7745fb907ebc86c94f1dbb02c0c1fd21b22f215f0582f1" exitCode=0 Oct 30 09:25:43.206066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.205778 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" event={"ID":"443609f7-d69a-413b-88d7-7cd36d9cafae","Type":"ContainerDied","Data":"d23b9418dcd6bde52f7745fb907ebc86c94f1dbb02c0c1fd21b22f215f0582f1"} Oct 30 09:25:43.209442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.209416 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" event={"ID":"7ec458ac-4f29-4839-9b86-62edb2d1bf69","Type":"ContainerStarted","Data":"fc83b96013f548945e90fd8ca9a618655413a11e69b3a9896c04e469f5418075"} Oct 30 09:25:43.209584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.209465 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" event={"ID":"7ec458ac-4f29-4839-9b86-62edb2d1bf69","Type":"ContainerStarted","Data":"02e3543c2fccf78c0027e89e91faf29f0010cdd2c4278fd2f701f815233e1a63"} Oct 30 09:25:43.258203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.258165 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:25:43.262190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.262158 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7d496775cb-78w6r"] Oct 30 09:25:43.296321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:43.296268 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" podStartSLOduration=2.2962543269999998 podStartE2EDuration="2.296254327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:43.295694256 +0000 UTC m=+1907.019121423" watchObservedRunningTime="2026-10-30 09:25:43.296254327 +0000 UTC m=+1907.019681503" Oct 30 09:25:44.221346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.221301 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" event={"ID":"8d4dd8b8-0a42-461e-90f2-e122b193bc7c","Type":"ContainerStarted","Data":"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1"} Oct 30 09:25:44.221750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.221725 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:44.224677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.224651 4920 generic.go:356] "Generic (PLEG): container finished" podID="7ec458ac-4f29-4839-9b86-62edb2d1bf69" containerID="fc83b96013f548945e90fd8ca9a618655413a11e69b3a9896c04e469f5418075" exitCode=0 Oct 30 09:25:44.224896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.224677 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" event={"ID":"7ec458ac-4f29-4839-9b86-62edb2d1bf69","Type":"ContainerDied","Data":"fc83b96013f548945e90fd8ca9a618655413a11e69b3a9896c04e469f5418075"} Oct 30 09:25:44.261778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.261714 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" podStartSLOduration=1.394787208 podStartE2EDuration="5.261697102s" totalImagesPullingTime="3.866909894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:44.260997333 +0000 UTC m=+1907.984424476" watchObservedRunningTime="2026-10-30 09:25:44.261697102 +0000 UTC m=+1907.985124246" Oct 30 09:25:44.403280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.403245 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7151869-fb12-484e-8ce0-eb25159807a7" path="/var/lib/kubelet/pods/a7151869-fb12-484e-8ce0-eb25159807a7/volumes" Oct 30 09:25:44.404243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.404201 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5038659-7424-4463-98ec-7b52cddefa4e" path="/var/lib/kubelet/pods/d5038659-7424-4463-98ec-7b52cddefa4e/volumes" Oct 30 09:25:44.414279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.414250 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:44.414534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.414511 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:44.422445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.422419 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\") pod \"443609f7-d69a-413b-88d7-7cd36d9cafae\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " Oct 30 09:25:44.422585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.422487 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\" (UniqueName: \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\") pod \"443609f7-d69a-413b-88d7-7cd36d9cafae\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " Oct 30 09:25:44.422692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.422671 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\") pod \"443609f7-d69a-413b-88d7-7cd36d9cafae\" (UID: \"443609f7-d69a-413b-88d7-7cd36d9cafae\") " Oct 30 09:25:44.423048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.423011 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca" pod "443609f7-d69a-413b-88d7-7cd36d9cafae" (UID: "443609f7-d69a-413b-88d7-7cd36d9cafae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:44.423590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.423565 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:44.424819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.424793 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca" pod "443609f7-d69a-413b-88d7-7cd36d9cafae" (UID: "443609f7-d69a-413b-88d7-7cd36d9cafae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:44.425170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.425138 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh" pod "443609f7-d69a-413b-88d7-7cd36d9cafae" (UID: "443609f7-d69a-413b-88d7-7cd36d9cafae"). InnerVolumeSpecName "kube-api-access-5xvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:44.526309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.525801 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvvh\" (UniqueName: \"kubernetes.io/projected/443609f7-d69a-413b-88d7-7cd36d9cafae-kube-api-access-5xvvh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:44.526309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:44.525845 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443609f7-d69a-413b-88d7-7cd36d9cafae-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:45.240148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.240118 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" Oct 30 09:25:45.240362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.240138 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg" event={"ID":"443609f7-d69a-413b-88d7-7cd36d9cafae","Type":"ContainerDied","Data":"6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0"} Oct 30 09:25:45.240362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.240254 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a176527b818c19cc93cdf8cd5e5619cda550c6c2efb9d17debcec952b817fa0" Oct 30 09:25:45.401852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.401822 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:45.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.437882 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\") pod \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " Oct 30 09:25:45.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.438128 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\") pod \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " Oct 30 09:25:45.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.438187 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\" (UniqueName: \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\") pod \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\" (UID: \"7ec458ac-4f29-4839-9b86-62edb2d1bf69\") " Oct 30 09:25:45.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.442795 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca" pod "7ec458ac-4f29-4839-9b86-62edb2d1bf69" (UID: "7ec458ac-4f29-4839-9b86-62edb2d1bf69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:45.446700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.446641 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf" pod "7ec458ac-4f29-4839-9b86-62edb2d1bf69" (UID: "7ec458ac-4f29-4839-9b86-62edb2d1bf69"). InnerVolumeSpecName "kube-api-access-2vwjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:45.451346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.451315 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca" pod "7ec458ac-4f29-4839-9b86-62edb2d1bf69" (UID: "7ec458ac-4f29-4839-9b86-62edb2d1bf69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:45.542956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.542919 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwjf\" (UniqueName: \"kubernetes.io/projected/7ec458ac-4f29-4839-9b86-62edb2d1bf69-kube-api-access-2vwjf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:45.542956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.542953 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:45.542956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:45.542965 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec458ac-4f29-4839-9b86-62edb2d1bf69-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.255233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.255182 4920 generic.go:356] "Generic (PLEG): container finished" podID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerID="bda354e62e5922babb734bf87526ffaa8cdd41ff466284f07734a99b709038f3" exitCode=0 Oct 30 09:25:46.255414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.255274 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerDied","Data":"bda354e62e5922babb734bf87526ffaa8cdd41ff466284f07734a99b709038f3"} Oct 30 09:25:46.255414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.255351 4920 scope.go:122] "RemoveContainer" containerID="285ee522f1efa87982c0e6f303dffa52fbdfc8206104fbbd16731ade030caace" Oct 30 09:25:46.257017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.256997 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" event={"ID":"7ec458ac-4f29-4839-9b86-62edb2d1bf69","Type":"ContainerDied","Data":"02e3543c2fccf78c0027e89e91faf29f0010cdd2c4278fd2f701f815233e1a63"} Oct 30 09:25:46.257130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.257043 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e3543c2fccf78c0027e89e91faf29f0010cdd2c4278fd2f701f815233e1a63" Oct 30 09:25:46.257130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.257060 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz" Oct 30 09:25:46.303717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.303688 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:25:46.352414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.352383 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.352584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.352476 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353286 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353446 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353482 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353553 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353577 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:46.353739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353664 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353704 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353768 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\") pod \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\" (UID: \"43bd0c51-fbe9-4c04-a360-18a60ae22ce7\") " Oct 30 09:25:46.353912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.353851 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:46.354061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354040 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:46.354324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354302 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:46.354428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354334 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:46.354601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354530 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.354601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354556 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.354601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354570 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.354924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354881 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.354924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.354900 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.356056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.355916 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "kube-api-access-hm6sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:46.356306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.356283 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:46.356966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.356714 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:46.358620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.358595 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token" pod "43bd0c51-fbe9-4c04-a360-18a60ae22ce7" (UID: "43bd0c51-fbe9-4c04-a360-18a60ae22ce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:46.446413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.446380 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:46.456792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.456761 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.456792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.456791 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6sb\" (UniqueName: \"kubernetes.io/projected/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-kube-api-access-hm6sb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.456983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.456808 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:46.456983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:46.456825 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bd0c51-fbe9-4c04-a360-18a60ae22ce7-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:47.266632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:47.266592 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" Oct 30 09:25:47.266814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:47.266626 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s" event={"ID":"43bd0c51-fbe9-4c04-a360-18a60ae22ce7","Type":"ContainerDied","Data":"11b7e6c10e792a2fc58ae673db711ceb3d5b4b2fa9ae392d93212bf969c93840"} Oct 30 09:25:47.266814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:47.266694 4920 scope.go:122] "RemoveContainer" containerID="bda354e62e5922babb734bf87526ffaa8cdd41ff466284f07734a99b709038f3" Oct 30 09:25:47.358491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:47.358459 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:25:47.363169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:47.363133 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-58fadf-openshift-default-778b9c9669-clb5s"] Oct 30 09:25:48.053633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.053591 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:48.162239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.158068 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:48.261954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.261916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:48.399623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.399540 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" path="/var/lib/kubelet/pods/43bd0c51-fbe9-4c04-a360-18a60ae22ce7/volumes" Oct 30 09:25:48.475909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.475876 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:25:48.476162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.476130 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" containerID="cri-o://ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b" gracePeriod=30 Oct 30 09:25:48.515484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:48.515453 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:49.088336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.088296 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp"] Oct 30 09:25:49.089142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089117 4920 cpu_manager.go:414] "containerMap: removing container" podUID="443609f7-d69a-413b-88d7-7cd36d9cafae" containerName="revoke-keys" Oct 30 09:25:49.089142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089142 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443609f7-d69a-413b-88d7-7cd36d9cafae" containerName="revoke-keys" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089165 4920 cpu_manager.go:414] "containerMap: removing container" podUID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" containerName="payload-processing" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089173 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" containerName="payload-processing" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089195 4920 cpu_manager.go:414] "containerMap: removing container" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089203 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089231 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7ec458ac-4f29-4839-9b86-62edb2d1bf69" containerName="revoke-keys" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089239 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec458ac-4f29-4839-9b86-62edb2d1bf69" containerName="revoke-keys" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089272 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a7151869-fb12-484e-8ce0-eb25159807a7" containerName="payload-processing" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089279 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7151869-fb12-484e-8ce0-eb25159807a7" containerName="payload-processing" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089324 4920 cpu_manager.go:414] "containerMap: removing container" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089331 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089520 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089541 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="443609f7-d69a-413b-88d7-7cd36d9cafae" containerName="revoke-keys" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089549 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31fd1df-5a8a-4dcd-940d-17aae905fc95" containerName="payload-processing" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089560 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd0c51-fbe9-4c04-a360-18a60ae22ce7" containerName="istio-proxy" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089568 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec458ac-4f29-4839-9b86-62edb2d1bf69" containerName="revoke-keys" Oct 30 09:25:49.089591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.089580 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7151869-fb12-484e-8ce0-eb25159807a7" containerName="payload-processing" Oct 30 09:25:49.095073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.095051 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.111856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.111828 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp"] Oct 30 09:25:49.143997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.143960 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:25:49.144202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.144001 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:25:49.209429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.209393 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.209686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.209663 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.209755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.209716 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xm46\" (UniqueName: \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.311572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.311539 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.311785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.311593 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xm46\" (UniqueName: \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.311785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.311636 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.312312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.312291 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.313929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.313904 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.330665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.330625 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xm46\" (UniqueName: \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\") pod \"maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.412525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.412432 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:49.597621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.597576 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:49.719938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.719846 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:49.730296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.730270 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp"] Oct 30 09:25:49.822096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:49.822058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:50.007840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.007767 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:50.180110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.179161 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:50.246405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.246378 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:25:50.311202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.311164 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" event={"ID":"2a478d34-4231-48b8-a080-b8639b5d119f","Type":"ContainerStarted","Data":"c30c7a4bce1baf59022a7fe044e9098d2cfc915f307be5097f10d4204e93543a"} Oct 30 09:25:50.311410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.311265 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" event={"ID":"2a478d34-4231-48b8-a080-b8639b5d119f","Type":"ContainerStarted","Data":"fa6b45919a524749bc2e85de83fcdd4c841c80f2298cfffebacc9476cdb0da0e"} Oct 30 09:25:50.352174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.352100 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" podStartSLOduration=1.352083408 podStartE2EDuration="1.352083408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:25:50.348558459 +0000 UTC m=+1914.071985602" watchObservedRunningTime="2026-10-30 09:25:50.352083408 +0000 UTC m=+1914.075510550" Oct 30 09:25:50.527181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.527147 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn"] Oct 30 09:25:50.531531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.531513 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.549778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.549745 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn"] Oct 30 09:25:50.631566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.631476 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.631566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.631551 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.631758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.631628 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgr9w\" (UniqueName: \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.732762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.732725 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.732939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.732778 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.732939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.732821 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgr9w\" (UniqueName: \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.733665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.733633 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.735137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.735112 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.751441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.751413 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgr9w\" (UniqueName: \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\") pod \"maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:50.847695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:50.847660 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:51.151550 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:25:51.151518 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc801574e_ea54_4fee_8a1d_ebf965958dbb.slice/crio-5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535 WatchSource:0}: Error finding container 5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535: Status 404 returned error can't find the container with id 5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535 Oct 30 09:25:51.153192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:51.153167 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn"] Oct 30 09:25:51.324352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:51.324314 4920 generic.go:356] "Generic (PLEG): container finished" podID="2a478d34-4231-48b8-a080-b8639b5d119f" containerID="c30c7a4bce1baf59022a7fe044e9098d2cfc915f307be5097f10d4204e93543a" exitCode=0 Oct 30 09:25:51.324835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:51.324399 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" event={"ID":"2a478d34-4231-48b8-a080-b8639b5d119f","Type":"ContainerDied","Data":"c30c7a4bce1baf59022a7fe044e9098d2cfc915f307be5097f10d4204e93543a"} Oct 30 09:25:51.326025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:51.325993 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" event={"ID":"c801574e-ea54-4fee-8a1d-ebf965958dbb","Type":"ContainerStarted","Data":"5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535"} Oct 30 09:25:51.814326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:51.814286 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:52.060347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.060309 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:25:52.192712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.192677 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:52.294068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.293442 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:52.336798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.336761 4920 generic.go:356] "Generic (PLEG): container finished" podID="c801574e-ea54-4fee-8a1d-ebf965958dbb" containerID="8c5cab9ef66cfcddf3df14083f57635edcd57bd18f524514b3b8a765a3ec1b84" exitCode=0 Oct 30 09:25:52.337375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.336845 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" event={"ID":"c801574e-ea54-4fee-8a1d-ebf965958dbb","Type":"ContainerDied","Data":"8c5cab9ef66cfcddf3df14083f57635edcd57bd18f524514b3b8a765a3ec1b84"} Oct 30 09:25:52.532065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.532037 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:52.660099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.660017 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\") pod \"2a478d34-4231-48b8-a080-b8639b5d119f\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " Oct 30 09:25:52.660306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.660184 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\" (UniqueName: \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\") pod \"2a478d34-4231-48b8-a080-b8639b5d119f\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " Oct 30 09:25:52.660369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.660316 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\") pod \"2a478d34-4231-48b8-a080-b8639b5d119f\" (UID: \"2a478d34-4231-48b8-a080-b8639b5d119f\") " Oct 30 09:25:52.660704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.660678 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca" pod "2a478d34-4231-48b8-a080-b8639b5d119f" (UID: "2a478d34-4231-48b8-a080-b8639b5d119f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:52.662368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.662344 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca" pod "2a478d34-4231-48b8-a080-b8639b5d119f" (UID: "2a478d34-4231-48b8-a080-b8639b5d119f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:52.662466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.662392 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46" pod "2a478d34-4231-48b8-a080-b8639b5d119f" (UID: "2a478d34-4231-48b8-a080-b8639b5d119f"). InnerVolumeSpecName "kube-api-access-5xm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:52.669980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.669946 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:25:52.670093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.670010 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:25:52.768447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.768359 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xm46\" (UniqueName: \"kubernetes.io/projected/2a478d34-4231-48b8-a080-b8639b5d119f-kube-api-access-5xm46\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:52.768447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.768402 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:52.768447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:52.768417 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a478d34-4231-48b8-a080-b8639b5d119f-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:53.347307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.347271 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" Oct 30 09:25:53.347307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.347287 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp" event={"ID":"2a478d34-4231-48b8-a080-b8639b5d119f","Type":"ContainerDied","Data":"fa6b45919a524749bc2e85de83fcdd4c841c80f2298cfffebacc9476cdb0da0e"} Oct 30 09:25:53.347830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.347344 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa6b45919a524749bc2e85de83fcdd4c841c80f2298cfffebacc9476cdb0da0e" Oct 30 09:25:53.496900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.496877 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:53.556240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.556189 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:53.683305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.683196 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\") pod \"c801574e-ea54-4fee-8a1d-ebf965958dbb\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " Oct 30 09:25:53.683305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.683260 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\") pod \"c801574e-ea54-4fee-8a1d-ebf965958dbb\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " Oct 30 09:25:53.683539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.683365 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\" (UniqueName: \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\") pod \"c801574e-ea54-4fee-8a1d-ebf965958dbb\" (UID: \"c801574e-ea54-4fee-8a1d-ebf965958dbb\") " Oct 30 09:25:53.684066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.684036 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca" pod "c801574e-ea54-4fee-8a1d-ebf965958dbb" (UID: "c801574e-ea54-4fee-8a1d-ebf965958dbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:53.686927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.686893 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca" pod "c801574e-ea54-4fee-8a1d-ebf965958dbb" (UID: "c801574e-ea54-4fee-8a1d-ebf965958dbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:53.687052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.686944 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w" pod "c801574e-ea54-4fee-8a1d-ebf965958dbb" (UID: "c801574e-ea54-4fee-8a1d-ebf965958dbb"). InnerVolumeSpecName "kube-api-access-hgr9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:53.700544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.700514 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:53.785502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.785468 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:53.785502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.785497 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c801574e-ea54-4fee-8a1d-ebf965958dbb-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:53.785502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.785510 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgr9w\" (UniqueName: \"kubernetes.io/projected/c801574e-ea54-4fee-8a1d-ebf965958dbb-kube-api-access-hgr9w\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:53.817261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:53.817188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:54.224774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.224743 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:25:54.298456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.298424 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.298681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.298473 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.298681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.298504 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.298681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.298544 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.298874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.298767 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:54.299387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.299355 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:54.299508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.299451 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.299881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.299859 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.300205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.300182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:54.300392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.300240 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.300620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.300592 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:54.300690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.300369 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.300802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.300784 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\") pod \"107feebf-f523-41d5-8543-3601ad9e152f\" (UID: \"107feebf-f523-41d5-8543-3601ad9e152f\") " Oct 30 09:25:54.301020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301000 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:54.301569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301541 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:54.301661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301572 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "kube-api-access-8p7ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:54.301718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301678 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301692 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107feebf-f523-41d5-8543-3601ad9e152f-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301701 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7ql\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-kube-api-access-8p7ql\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301712 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301725 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301736 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.301918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.301744 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107feebf-f523-41d5-8543-3601ad9e152f-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.303395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.303363 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:54.303642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.303613 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo" pod "107feebf-f523-41d5-8543-3601ad9e152f" (UID: "107feebf-f523-41d5-8543-3601ad9e152f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:54.359268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.359193 4920 generic.go:356] "Generic (PLEG): container finished" podID="107feebf-f523-41d5-8543-3601ad9e152f" containerID="ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b" exitCode=0 Oct 30 09:25:54.359268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.359272 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" Oct 30 09:25:54.359769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.359262 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerDied","Data":"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b"} Oct 30 09:25:54.359769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.359413 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p" event={"ID":"107feebf-f523-41d5-8543-3601ad9e152f","Type":"ContainerDied","Data":"b3c45322913a6a2a36f41d3de3f2f0db000d843fc10b389a391d540ef38b60c8"} Oct 30 09:25:54.359769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.359447 4920 scope.go:122] "RemoveContainer" containerID="ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b" Oct 30 09:25:54.360961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.360887 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" Oct 30 09:25:54.360961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.360906 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn" event={"ID":"c801574e-ea54-4fee-8a1d-ebf965958dbb","Type":"ContainerDied","Data":"5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535"} Oct 30 09:25:54.360961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.360943 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de3cb463a40f8ac3f062052206f836555313ad5c12ac74f0897de426c36a535" Oct 30 09:25:54.374096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.374077 4920 scope.go:122] "RemoveContainer" containerID="20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f" Oct 30 09:25:54.384817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.384798 4920 scope.go:122] "RemoveContainer" containerID="ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b" Oct 30 09:25:54.385096 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:54.385076 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b\": container with ID starting with ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b not found: ID does not exist" containerID="ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b" Oct 30 09:25:54.385176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.385110 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b"} err="failed to get container status \"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b\": rpc error: code = NotFound desc = could not find container \"ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b\": container with ID starting with ed11bb9802401f6475ba66be8c9beafd8f3cf61b1bf2a39caadcb23da1c9a51b not found: ID does not exist" Oct 30 09:25:54.385176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.385138 4920 scope.go:122] "RemoveContainer" containerID="20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f" Oct 30 09:25:54.385491 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:54.385475 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f\": container with ID starting with 20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f not found: ID does not exist" containerID="20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f" Oct 30 09:25:54.385530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.385497 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f"} err="failed to get container status \"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f\": rpc error: code = NotFound desc = could not find container \"20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f\": container with ID starting with 20a1cf149f159f19c757a0330b74c57ae30f9b1cc3d5f19a4d8c77f96b16e65f not found: ID does not exist" Oct 30 09:25:54.391387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.391366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:54.402855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.402829 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107feebf-f523-41d5-8543-3601ad9e152f-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.402963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.402858 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107feebf-f523-41d5-8543-3601ad9e152f-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.412677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.412654 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:25:54.416746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.416723 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f40a3-openshift-default-ff78d497f-sqr9p"] Oct 30 09:25:54.466045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.466011 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg"] Oct 30 09:25:54.469397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.469374 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cbe974-r4llg"] Oct 30 09:25:54.512374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.512335 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:54.917717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:54.917621 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:56.181036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:56.181004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:56.409401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:56.409366 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107feebf-f523-41d5-8543-3601ad9e152f" path="/var/lib/kubelet/pods/107feebf-f523-41d5-8543-3601ad9e152f/volumes" Oct 30 09:25:56.409874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:56.409859 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443609f7-d69a-413b-88d7-7cd36d9cafae" path="/var/lib/kubelet/pods/443609f7-d69a-413b-88d7-7cd36d9cafae/volumes" Oct 30 09:25:57.060906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.060863 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" podUID="bc7a641a-6b65-4292-afb6-96a94411ca49" containerName="payload-processing" containerID="cri-o://71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696" gracePeriod=25 Oct 30 09:25:57.366160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.366132 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:25:57.388777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.388735 4920 generic.go:356] "Generic (PLEG): container finished" podID="bc7a641a-6b65-4292-afb6-96a94411ca49" containerID="71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696" exitCode=0 Oct 30 09:25:57.388967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.388790 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" Oct 30 09:25:57.388967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.388826 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" event={"ID":"bc7a641a-6b65-4292-afb6-96a94411ca49","Type":"ContainerDied","Data":"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696"} Oct 30 09:25:57.388967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.388882 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj" event={"ID":"bc7a641a-6b65-4292-afb6-96a94411ca49","Type":"ContainerDied","Data":"87ccf29eb60d939dc4ea3bd847f144aa8b43a990b931bc5f3abaf4a99b919b35"} Oct 30 09:25:57.388967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.388907 4920 scope.go:122] "RemoveContainer" containerID="71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696" Oct 30 09:25:57.400392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.400371 4920 scope.go:122] "RemoveContainer" containerID="71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696" Oct 30 09:25:57.400750 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:25:57.400726 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696\": container with ID starting with 71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696 not found: ID does not exist" containerID="71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696" Oct 30 09:25:57.400856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.400757 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696"} err="failed to get container status \"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696\": rpc error: code = NotFound desc = could not find container \"71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696\": container with ID starting with 71fda5de2dd513190c3bd1813b18c3d84eb8e973dfdf5c86438ca453687f0696 not found: ID does not exist" Oct 30 09:25:57.447165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.447129 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\") pod \"bc7a641a-6b65-4292-afb6-96a94411ca49\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " Oct 30 09:25:57.447401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.447257 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\" (UniqueName: \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\") pod \"bc7a641a-6b65-4292-afb6-96a94411ca49\" (UID: \"bc7a641a-6b65-4292-afb6-96a94411ca49\") " Oct 30 09:25:57.447541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.447515 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume" pod "bc7a641a-6b65-4292-afb6-96a94411ca49" (UID: "bc7a641a-6b65-4292-afb6-96a94411ca49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:57.447901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.447880 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7a641a-6b65-4292-afb6-96a94411ca49-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:57.449469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.449440 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp" pod "bc7a641a-6b65-4292-afb6-96a94411ca49" (UID: "bc7a641a-6b65-4292-afb6-96a94411ca49"). InnerVolumeSpecName "kube-api-access-7hqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:57.549180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.549138 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqpp\" (UniqueName: \"kubernetes.io/projected/bc7a641a-6b65-4292-afb6-96a94411ca49-kube-api-access-7hqpp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:25:57.736712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.736672 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:25:57.746281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.746247 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9371b7-789f894498-rb2gj"] Oct 30 09:25:57.836114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:57.836048 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:25:58.399055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:58.399020 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7a641a-6b65-4292-afb6-96a94411ca49" path="/var/lib/kubelet/pods/bc7a641a-6b65-4292-afb6-96a94411ca49/volumes" Oct 30 09:25:59.872099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:25:59.872061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:00.145440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.145353 4920 scope.go:122] "RemoveContainer" containerID="d2087846ba74fcbb10537699f9fdb3be0407e6a0ba98fe51c03c987b23d953ca" Oct 30 09:26:00.157515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.157492 4920 scope.go:122] "RemoveContainer" containerID="1b1b7d4c48119d5ce2d9d231c94096d5f134be3a3acb1d2e4f9fdd4c602f4878" Oct 30 09:26:00.170418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.170387 4920 scope.go:122] "RemoveContainer" containerID="c9f51f560bcd007d951f8ab9c7fb3e77c0f289546b4f5026a954350d51e79f8c" Oct 30 09:26:00.182333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.182303 4920 scope.go:122] "RemoveContainer" containerID="91d4734f30aa9b9080df9b5e3fcfde07cc4a078ad628c77da8a9fea2475c0568" Oct 30 09:26:00.193459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.193435 4920 scope.go:122] "RemoveContainer" containerID="cf2a2bc524a94967a69fbfc2bb3156a8fbb48164eb7173d9a4b738494aa2c08b" Oct 30 09:26:00.204137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:00.204115 4920 scope.go:122] "RemoveContainer" containerID="fe13fd8945a838281e529c9109fc4a625c3f27b82ac3040531cd233fac6bdc6c" Oct 30 09:26:01.141499 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:01.141456 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:26:01.614241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.602741 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:01.711240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.708322 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:01.819240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.818743 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:01.943318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.936294 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:01.952713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.952674 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:26:01.953148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:01.953122 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" containerID="cri-o://9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730" gracePeriod=30 Oct 30 09:26:03.523242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.522728 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:03.642526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.642489 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.938778 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939686 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bc7a641a-6b65-4292-afb6-96a94411ca49" containerName="payload-processing" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939707 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7a641a-6b65-4292-afb6-96a94411ca49" containerName="payload-processing" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939731 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2a478d34-4231-48b8-a080-b8639b5d119f" containerName="revoke-keys" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939742 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a478d34-4231-48b8-a080-b8639b5d119f" containerName="revoke-keys" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939759 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c801574e-ea54-4fee-8a1d-ebf965958dbb" containerName="revoke-keys" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939767 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c801574e-ea54-4fee-8a1d-ebf965958dbb" containerName="revoke-keys" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939797 4920 cpu_manager.go:414] "containerMap: removing container" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939805 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939963 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939985 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7a641a-6b65-4292-afb6-96a94411ca49" containerName="payload-processing" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.939998 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a478d34-4231-48b8-a080-b8639b5d119f" containerName="revoke-keys" Oct 30 09:26:03.940637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.940010 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c801574e-ea54-4fee-8a1d-ebf965958dbb" containerName="revoke-keys" Oct 30 09:26:03.951818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.951790 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.966510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.966482 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:26:03.969687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.969658 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c9232f-openshift-default-dockercfg-w7jlr\"" Oct 30 09:26:03.980279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980250 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.980426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980362 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.980508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980484 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.980594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980580 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.981147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980798 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.981147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980837 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.981147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980894 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.981147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.980955 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.981147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.981011 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtmx\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:03.982052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:03.982016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:04.103044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.103006 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.104586 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105318 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105368 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105430 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105499 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105562 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtmx\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105665 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105707 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.105771 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.106114 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.111244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.107552 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.143300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.137142 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.152664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.152612 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.166040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.165817 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.174774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.173871 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.174774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.174479 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:04.215359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.212723 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtmx\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.237295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.222805 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.272619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.272575 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:04.643052 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:04.643021 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4445a69_4edc_4ffb_b230_0d97bfc7ea9b.slice/crio-077098f57850116e54be2841597dd56f79f86a1e1ce354892367ef1045466005 WatchSource:0}: Error finding container 077098f57850116e54be2841597dd56f79f86a1e1ce354892367ef1045466005: Status 404 returned error can't find the container with id 077098f57850116e54be2841597dd56f79f86a1e1ce354892367ef1045466005 Oct 30 09:26:04.645565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.645529 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:04.645678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.645600 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:04.645678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.645628 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:04.646322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:04.646295 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:26:05.038750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.038716 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2"] Oct 30 09:26:05.046388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.046356 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e035ff14-2mqf2"] Oct 30 09:26:05.192962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.188778 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:05.350699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.350546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:05.435746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.435707 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:05.524175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.524137 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerStarted","Data":"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc"} Oct 30 09:26:05.524446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.524425 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerStarted","Data":"077098f57850116e54be2841597dd56f79f86a1e1ce354892367ef1045466005"} Oct 30 09:26:05.755917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.733076 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podStartSLOduration=2.733054902 podStartE2EDuration="2.733054902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:05.584854179 +0000 UTC m=+1929.308281343" watchObservedRunningTime="2026-10-30 09:26:05.733054902 +0000 UTC m=+1929.456482047" Oct 30 09:26:05.836276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.836025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:05.938502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:05.936108 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:06.063471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.061371 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn"] Oct 30 09:26:06.063471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.063386 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-962e2e-m2lmn"] Oct 30 09:26:06.223944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.213183 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:06.275200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.273642 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:06.284019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.283990 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:06.410831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.410746 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9c0e23-6b99-4f81-a475-85a90dda3157" path="/var/lib/kubelet/pods/3a9c0e23-6b99-4f81-a475-85a90dda3157/volumes" Oct 30 09:26:06.411840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.411817 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c801574e-ea54-4fee-8a1d-ebf965958dbb" path="/var/lib/kubelet/pods/c801574e-ea54-4fee-8a1d-ebf965958dbb/volumes" Oct 30 09:26:06.537276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.537245 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:06.538360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.538336 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:26:06.539713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:06.539642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:07.267070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.267041 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:26:07.299860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.299818 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.299880 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.299925 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.299962 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.299997 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300118 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:07.300185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300124 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300349 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300398 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:07.300496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300409 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300460 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\") pod \"e0a73090-2cee-41c2-8873-492b7bfd297d\" (UID: \"e0a73090-2cee-41c2-8873-492b7bfd297d\") " Oct 30 09:26:07.300663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300594 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:07.300889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300858 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:07.300994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.300939 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:07.301079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.301047 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.301079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.301064 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a73090-2cee-41c2-8873-492b7bfd297d-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.301259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.301079 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.301259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.301092 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.301259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.301105 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.303858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.303819 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:07.304149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.304101 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:26:07.304572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.304547 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "kube-api-access-9hsnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:07.309302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.309276 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token" pod "e0a73090-2cee-41c2-8873-492b7bfd297d" (UID: "e0a73090-2cee-41c2-8873-492b7bfd297d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:07.402429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.402388 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hsnv\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-kube-api-access-9hsnv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.402429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.402425 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.402714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.402440 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.402714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.402454 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a73090-2cee-41c2-8873-492b7bfd297d-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:07.548159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.548123 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerID="9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730" exitCode=0 Oct 30 09:26:07.548357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.548193 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" Oct 30 09:26:07.548357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.548201 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerDied","Data":"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730"} Oct 30 09:26:07.548357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.548317 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd" event={"ID":"e0a73090-2cee-41c2-8873-492b7bfd297d","Type":"ContainerDied","Data":"9a8502bf5b6d6d2a542b0793a366e1cbc5e62a99f9b9ff12cbc9eb57eecc0ea7"} Oct 30 09:26:07.548357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.548337 4920 scope.go:122] "RemoveContainer" containerID="9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730" Oct 30 09:26:07.566696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.566673 4920 scope.go:122] "RemoveContainer" containerID="44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026" Oct 30 09:26:07.580551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.580529 4920 scope.go:122] "RemoveContainer" containerID="9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730" Oct 30 09:26:07.580920 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:07.580894 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730\": container with ID starting with 9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730 not found: ID does not exist" containerID="9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730" Oct 30 09:26:07.580975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.580934 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730"} err="failed to get container status \"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730\": rpc error: code = NotFound desc = could not find container \"9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730\": container with ID starting with 9676764b241ba3f43b6e4cdc78e521be656aa27e55e082d27db6c43088c1c730 not found: ID does not exist" Oct 30 09:26:07.580975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.580961 4920 scope.go:122] "RemoveContainer" containerID="44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026" Oct 30 09:26:07.581260 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:07.581237 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026\": container with ID starting with 44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026 not found: ID does not exist" containerID="44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026" Oct 30 09:26:07.581326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.581272 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026"} err="failed to get container status \"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026\": rpc error: code = NotFound desc = could not find container \"44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026\": container with ID starting with 44d5289e6761cfe2f080dd80dfcbab3db6103a8a1fecd9059eb6709690b2f026 not found: ID does not exist" Oct 30 09:26:07.617625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.617526 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:26:07.621025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.620998 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbe974-openshift-default-69dc69ff6f-7w7zd"] Oct 30 09:26:07.965809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:07.965694 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:08.065767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:08.065636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:08.194698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:08.194656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:08.403163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:08.403073 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" path="/var/lib/kubelet/pods/e0a73090-2cee-41c2-8873-492b7bfd297d/volumes" Oct 30 09:26:08.925003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:08.924969 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:09.947671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:09.947635 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:10.052036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.052005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:10.096245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096184 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:26:10.096781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096767 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.096826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096782 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.096826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096803 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.096826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096809 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.096922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096827 4920 cpu_manager.go:414] "containerMap: removing container" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:10.096922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096832 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:10.096922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096919 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.097008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096931 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a73090-2cee-41c2-8873-492b7bfd297d" containerName="istio-proxy" Oct 30 09:26:10.097008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.096941 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="107feebf-f523-41d5-8543-3601ad9e152f" containerName="istio-proxy" Oct 30 09:26:10.102300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.102274 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.140799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.126489 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:26:10.140972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.140948 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c9232f\"" Oct 30 09:26:10.141030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.140962 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c9232f-dockercfg-kjvcg\"" Oct 30 09:26:10.144639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.144609 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcwc\" (UniqueName: \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.144792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.144706 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.273535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.272023 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcwc\" (UniqueName: \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.273535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.272167 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.273535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.273104 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.308314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.306272 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcwc\" (UniqueName: \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\") pod \"payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.319560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.319505 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:10.350597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.350420 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:26:10.355448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.355423 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:26:10.355767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.355705 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.425325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.425100 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:10.479050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.479023 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.479408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.479389 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68pm\" (UniqueName: \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.577659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.577569 4920 generic.go:356] "Generic (PLEG): container finished" podID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerID="a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8" exitCode=0 Oct 30 09:26:10.577659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.577643 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerDied","Data":"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8"} Oct 30 09:26:10.578200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.578180 4920 scope.go:122] "RemoveContainer" containerID="a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8" Oct 30 09:26:10.580136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.580102 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:10.580290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.580188 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:10.580290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.580252 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:10.581264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.581239 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68pm\" (UniqueName: \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.581489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.581469 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.582333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.582311 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.611628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.611550 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68pm\" (UniqueName: \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\") pod \"payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.693526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.693496 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:10.763263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:10.763232 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:26:11.022023 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:11.021993 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0538287e_480f_4c4b_98f8_ce86b0b0fa43.slice/crio-a480d3f981afef42e89549a4c2924193c4307f58c764254e00b54b300f1aabee WatchSource:0}: Error finding container a480d3f981afef42e89549a4c2924193c4307f58c764254e00b54b300f1aabee: Status 404 returned error can't find the container with id a480d3f981afef42e89549a4c2924193c4307f58c764254e00b54b300f1aabee Oct 30 09:26:11.026920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.026885 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:26:11.161422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.161394 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd"] Oct 30 09:26:11.164541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.164517 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.179292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.179266 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:26:11.179418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.179402 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:26:11.183854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.183827 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd"] Oct 30 09:26:11.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.188547 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.188668 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.188793 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsn8\" (UniqueName: \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.289659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.289625 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.289868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.289766 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsn8\" (UniqueName: \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.289939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.289897 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.290473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.290445 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.292372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.292348 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.318245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.317874 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsn8\" (UniqueName: \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\") pod \"maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.477980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.477883 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:11.643243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.631624 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" event={"ID":"4dd4f29d-c23f-434b-a52d-c30df3f9618e","Type":"ContainerStarted","Data":"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96"} Oct 30 09:26:11.643243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.631688 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" event={"ID":"4dd4f29d-c23f-434b-a52d-c30df3f9618e","Type":"ContainerStarted","Data":"e25b46a77cd2fa3083a7265630746169450d7ceeb4966d481d816137a17dbef1"} Oct 30 09:26:11.644001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.643951 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:11.646628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.646589 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" event={"ID":"0538287e-480f-4c4b-98f8-ce86b0b0fa43","Type":"ContainerStarted","Data":"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682"} Oct 30 09:26:11.646778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.646664 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" event={"ID":"0538287e-480f-4c4b-98f8-ce86b0b0fa43","Type":"ContainerStarted","Data":"a480d3f981afef42e89549a4c2924193c4307f58c764254e00b54b300f1aabee"} Oct 30 09:26:11.651143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.650430 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerStarted","Data":"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105"} Oct 30 09:26:11.683749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.683678 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" podStartSLOduration=1.683660992 podStartE2EDuration="1.683660992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:11.681208652 +0000 UTC m=+1935.404635795" watchObservedRunningTime="2026-10-30 09:26:11.683660992 +0000 UTC m=+1935.407088135" Oct 30 09:26:11.738310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.722297 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" podStartSLOduration=1.72227759 podStartE2EDuration="1.72227759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:11.719693027 +0000 UTC m=+1935.443120198" watchObservedRunningTime="2026-10-30 09:26:11.72227759 +0000 UTC m=+1935.445704736" Oct 30 09:26:11.837243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:11.832273 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd"] Oct 30 09:26:11.837788 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:11.837758 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9880b5e6_1079_4eed_afd7_f576879f5597.slice/crio-60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459 WatchSource:0}: Error finding container 60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459: Status 404 returned error can't find the container with id 60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459 Oct 30 09:26:12.160565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.160533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:12.223613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.223577 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:26:12.289616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.289518 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:26:12.291864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.291822 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:26:12.292021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.291889 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:26:12.660856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.660758 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" event={"ID":"9880b5e6-1079-4eed-afd7-f576879f5597","Type":"ContainerStarted","Data":"8fa49b40fc4ed5d7dc5b6c46f71a6fc708c8084ba55934aab83150bbc8b0006e"} Oct 30 09:26:12.661100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.660857 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" event={"ID":"9880b5e6-1079-4eed-afd7-f576879f5597","Type":"ContainerStarted","Data":"60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459"} Oct 30 09:26:12.661173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.661139 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:12.728584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:12.707072 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" podStartSLOduration=1.707050044 podStartE2EDuration="1.707050044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:12.704664442 +0000 UTC m=+1936.428091612" watchObservedRunningTime="2026-10-30 09:26:12.707050044 +0000 UTC m=+1936.430477189" Oct 30 09:26:13.015778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.015680 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:13.126825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.126787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:13.289280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.289251 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:26:13.289665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.289306 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:26:13.376268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.376237 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:13.670866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.670759 4920 generic.go:356] "Generic (PLEG): container finished" podID="9880b5e6-1079-4eed-afd7-f576879f5597" containerID="8fa49b40fc4ed5d7dc5b6c46f71a6fc708c8084ba55934aab83150bbc8b0006e" exitCode=0 Oct 30 09:26:13.670866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.670847 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" event={"ID":"9880b5e6-1079-4eed-afd7-f576879f5597","Type":"ContainerDied","Data":"8fa49b40fc4ed5d7dc5b6c46f71a6fc708c8084ba55934aab83150bbc8b0006e"} Oct 30 09:26:13.971764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:13.971667 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:14.289906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.289869 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:26:14.290319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.289926 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:26:14.879682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.879656 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:14.923815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.923711 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:14.977669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.977557 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\") pod \"9880b5e6-1079-4eed-afd7-f576879f5597\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " Oct 30 09:26:14.977841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.977673 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\" (UniqueName: \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\") pod \"9880b5e6-1079-4eed-afd7-f576879f5597\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " Oct 30 09:26:14.977936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.977861 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\") pod \"9880b5e6-1079-4eed-afd7-f576879f5597\" (UID: \"9880b5e6-1079-4eed-afd7-f576879f5597\") " Oct 30 09:26:14.978199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.978172 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca" pod "9880b5e6-1079-4eed-afd7-f576879f5597" (UID: "9880b5e6-1079-4eed-afd7-f576879f5597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:14.978590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.978564 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:14.979786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.979763 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca" pod "9880b5e6-1079-4eed-afd7-f576879f5597" (UID: "9880b5e6-1079-4eed-afd7-f576879f5597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:14.979999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:14.979981 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8" pod "9880b5e6-1079-4eed-afd7-f576879f5597" (UID: "9880b5e6-1079-4eed-afd7-f576879f5597"). InnerVolumeSpecName "kube-api-access-9nsn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:15.079605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.079570 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9880b5e6-1079-4eed-afd7-f576879f5597-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:15.079605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.079603 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsn8\" (UniqueName: \"kubernetes.io/projected/9880b5e6-1079-4eed-afd7-f576879f5597-kube-api-access-9nsn8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:15.289582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.289549 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:26:15.289737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.289595 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:26:15.357467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.357422 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:26:15.358521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.358496 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9880b5e6-1079-4eed-afd7-f576879f5597" containerName="revoke-keys" Oct 30 09:26:15.358677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.358652 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9880b5e6-1079-4eed-afd7-f576879f5597" containerName="revoke-keys" Oct 30 09:26:15.359043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.359029 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9880b5e6-1079-4eed-afd7-f576879f5597" containerName="revoke-keys" Oct 30 09:26:15.364239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.364054 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.374252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.372569 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:15.383228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.383164 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:26:15.394258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.392423 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-19a563-openshift-default-dockercfg-mz4s6\"" Oct 30 09:26:15.472268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.472231 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:26:15.472484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.472456 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" podUID="1c33b203-a7aa-482b-b211-2d597f4229e2" containerName="payload-processing" containerID="cri-o://33baf38ed2ea1f27b58dace9dae6187ca8c66acd34d4b93d09a6cb6bd7f52699" gracePeriod=30 Oct 30 09:26:15.481336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.481297 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:15.487111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.487081 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.487403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.487380 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.487563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.487544 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.487719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.487701 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cpj\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.487881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.487867 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.488026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.488012 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.488191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.488176 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.488627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.488599 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.488705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.488650 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.528253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.525020 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:26:15.528253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.525278 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" podUID="ae1951fb-37b2-4dae-b970-5ee244546a14" containerName="payload-pre-processing" containerID="cri-o://e20b02aab53b82ffc56d76b7d3ed1c67b1a19ddf9baaa76df26d7ae7da311439" gracePeriod=30 Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.590930 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.590995 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591029 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591193 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591264 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591301 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591359 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cpj\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591440 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.591492 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.592061 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.593280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.592572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.594008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.593870 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.594008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.593956 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.596757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.596725 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.627241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.597260 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.635859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.635820 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.649844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.649777 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.656328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.655793 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cpj\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\") pod \"e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.689057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.688753 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:15.718838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.697193 4920 generic.go:356] "Generic (PLEG): container finished" podID="ae1951fb-37b2-4dae-b970-5ee244546a14" containerID="e20b02aab53b82ffc56d76b7d3ed1c67b1a19ddf9baaa76df26d7ae7da311439" exitCode=0 Oct 30 09:26:15.718838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.697430 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" event={"ID":"ae1951fb-37b2-4dae-b970-5ee244546a14","Type":"ContainerDied","Data":"e20b02aab53b82ffc56d76b7d3ed1c67b1a19ddf9baaa76df26d7ae7da311439"} Oct 30 09:26:15.718838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.716189 4920 generic.go:356] "Generic (PLEG): container finished" podID="1c33b203-a7aa-482b-b211-2d597f4229e2" containerID="33baf38ed2ea1f27b58dace9dae6187ca8c66acd34d4b93d09a6cb6bd7f52699" exitCode=0 Oct 30 09:26:15.718838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.716313 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" event={"ID":"1c33b203-a7aa-482b-b211-2d597f4229e2","Type":"ContainerDied","Data":"33baf38ed2ea1f27b58dace9dae6187ca8c66acd34d4b93d09a6cb6bd7f52699"} Oct 30 09:26:15.736242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.729915 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" event={"ID":"9880b5e6-1079-4eed-afd7-f576879f5597","Type":"ContainerDied","Data":"60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459"} Oct 30 09:26:15.736242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.729981 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b1d41f1bd2ccd4a3a98ccbeeec33db2e4ecbc9afc2fb3c58a4e27673c77459" Oct 30 09:26:15.736242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.730077 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd" Oct 30 09:26:15.783835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:15.783797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:16.150242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.150111 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:16.166124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.166091 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:26:16.169653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.169623 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:26:16.248104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.248067 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:16.289981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.289940 4920 patch_prober.go:29] interesting pod/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 09:26:16.290199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.290004 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 09:26:16.295797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.295736 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp"] Oct 30 09:26:16.330899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.330876 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-356553-4r7tp"] Oct 30 09:26:16.336232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.336189 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:26:16.337370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.337338 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:16.337468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.337412 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:16.337468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.337458 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:16.347290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.347131 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:16.362263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.362232 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\" (UniqueName: \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\") pod \"ae1951fb-37b2-4dae-b970-5ee244546a14\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " Oct 30 09:26:16.362775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.362474 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\") pod \"1c33b203-a7aa-482b-b211-2d597f4229e2\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " Oct 30 09:26:16.362775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.362560 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\" (UniqueName: \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\") pod \"1c33b203-a7aa-482b-b211-2d597f4229e2\" (UID: \"1c33b203-a7aa-482b-b211-2d597f4229e2\") " Oct 30 09:26:16.362775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.362675 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\") pod \"ae1951fb-37b2-4dae-b970-5ee244546a14\" (UID: \"ae1951fb-37b2-4dae-b970-5ee244546a14\") " Oct 30 09:26:16.363166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.363140 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume" pod "ae1951fb-37b2-4dae-b970-5ee244546a14" (UID: "ae1951fb-37b2-4dae-b970-5ee244546a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:16.363166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.363159 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume" pod "1c33b203-a7aa-482b-b211-2d597f4229e2" (UID: "1c33b203-a7aa-482b-b211-2d597f4229e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:16.363381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.363317 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1951fb-37b2-4dae-b970-5ee244546a14-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:16.363381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.363338 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c33b203-a7aa-482b-b211-2d597f4229e2-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:16.364978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.364948 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls" pod "ae1951fb-37b2-4dae-b970-5ee244546a14" (UID: "ae1951fb-37b2-4dae-b970-5ee244546a14"). InnerVolumeSpecName "kube-api-access-pwkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:16.366067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.366047 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh" pod "1c33b203-a7aa-482b-b211-2d597f4229e2" (UID: "1c33b203-a7aa-482b-b211-2d597f4229e2"). InnerVolumeSpecName "kube-api-access-xlxvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:16.463309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.463121 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a478d34-4231-48b8-a080-b8639b5d119f" path="/var/lib/kubelet/pods/2a478d34-4231-48b8-a080-b8639b5d119f/volumes" Oct 30 09:26:16.465417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.464244 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkls\" (UniqueName: \"kubernetes.io/projected/ae1951fb-37b2-4dae-b970-5ee244546a14-kube-api-access-pwkls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:16.465417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.464273 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxvh\" (UniqueName: \"kubernetes.io/projected/1c33b203-a7aa-482b-b211-2d597f4229e2-kube-api-access-xlxvh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:16.746290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.745857 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" Oct 30 09:26:16.746290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.745856 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz" event={"ID":"1c33b203-a7aa-482b-b211-2d597f4229e2","Type":"ContainerDied","Data":"457bdf783aa7b90ec81bad21cfc032088963a08d5efb5fa993ea4d1acae4486b"} Oct 30 09:26:16.746290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.746066 4920 scope.go:122] "RemoveContainer" containerID="33baf38ed2ea1f27b58dace9dae6187ca8c66acd34d4b93d09a6cb6bd7f52699" Oct 30 09:26:16.748736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.748615 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerStarted","Data":"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6"} Oct 30 09:26:16.748736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.748678 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerStarted","Data":"5446b815202f7864288b21ea1da8205ce2b2f1c384850364e9ed771864289cba"} Oct 30 09:26:16.752006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.751982 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" Oct 30 09:26:16.752785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.751997 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt" event={"ID":"ae1951fb-37b2-4dae-b970-5ee244546a14","Type":"ContainerDied","Data":"5e392b6dce33506549296a6e0cfa0bf5b3bc08450ef0415e5521c99dc88172a3"} Oct 30 09:26:16.765688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.765663 4920 scope.go:122] "RemoveContainer" containerID="e20b02aab53b82ffc56d76b7d3ed1c67b1a19ddf9baaa76df26d7ae7da311439" Oct 30 09:26:16.778477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.778440 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:16.803662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.803625 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:26:16.810279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.810240 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-55d49b5bc-z6zdt"] Oct 30 09:26:16.842846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.842799 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:26:16.848836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.848801 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-5f76957cdb-tr5vz"] Oct 30 09:26:16.884823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:16.884744 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podStartSLOduration=1.8847249499999998 podStartE2EDuration="1.88472495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:16.88264318 +0000 UTC m=+1940.606070324" watchObservedRunningTime="2026-10-30 09:26:16.88472495 +0000 UTC m=+1940.608152094" Oct 30 09:26:17.294541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.294509 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:26:17.295574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.295551 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:26:17.689649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.689564 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:17.695517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.695494 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:17.771422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.771386 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:17.772412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:17.772385 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:26:18.185416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:18.185375 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:18.398734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:18.398690 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c33b203-a7aa-482b-b211-2d597f4229e2" path="/var/lib/kubelet/pods/1c33b203-a7aa-482b-b211-2d597f4229e2/volumes" Oct 30 09:26:18.399329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:18.399310 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1951fb-37b2-4dae-b970-5ee244546a14" path="/var/lib/kubelet/pods/ae1951fb-37b2-4dae-b970-5ee244546a14/volumes" Oct 30 09:26:19.219923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:19.219887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:19.338984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:19.338948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:19.969781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:19.969749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:20.903952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:20.903916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:21.011407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.011315 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:21.094918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.094879 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:26:21.095512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095496 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ae1951fb-37b2-4dae-b970-5ee244546a14" containerName="payload-pre-processing" Oct 30 09:26:21.095581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095513 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1951fb-37b2-4dae-b970-5ee244546a14" containerName="payload-pre-processing" Oct 30 09:26:21.095581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095541 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1c33b203-a7aa-482b-b211-2d597f4229e2" containerName="payload-processing" Oct 30 09:26:21.095581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095547 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c33b203-a7aa-482b-b211-2d597f4229e2" containerName="payload-processing" Oct 30 09:26:21.095684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095654 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1951fb-37b2-4dae-b970-5ee244546a14" containerName="payload-pre-processing" Oct 30 09:26:21.095684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.095665 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c33b203-a7aa-482b-b211-2d597f4229e2" containerName="payload-processing" Oct 30 09:26:21.099464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.099440 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.121166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.121135 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:26:21.124099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.124061 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.124343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.124315 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8gz\" (UniqueName: \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.127073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.126905 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-19a563\"" Oct 30 09:26:21.226296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.217646 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-19a563-dockercfg-lnfrx\"" Oct 30 09:26:21.247240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.244664 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8gz\" (UniqueName: \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.247240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.244797 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.247240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.245677 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.266256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.263679 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:26:21.296237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.294891 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8gz\" (UniqueName: \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\") pod \"payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.321348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.307132 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:26:21.321348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.307275 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.350108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.350076 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.350300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.350269 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfc88\" (UniqueName: \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.415928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.415892 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:21.452732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.452704 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.453176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.453149 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfc88\" (UniqueName: \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.453575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.453553 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.472757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.472725 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfc88\" (UniqueName: \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\") pod \"payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.641236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.633413 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:21.657018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.656986 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:21.737686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.736500 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:26:21.745704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.744864 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:21.828952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:21.828915 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" event={"ID":"7bca8563-e142-4c4d-852d-f68e56057a64","Type":"ContainerStarted","Data":"211e193be767ab63916ae7ca79f4d1ab12c328c16160cfcddcb7f918be2e4a32"} Oct 30 09:26:22.023542 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:22.023505 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5bbb9b_3dc4_4795_9b56_17ae03add46c.slice/crio-886c90c77d8708408992548141e0292115c096e95d2cbb7bc3773d2289479947 WatchSource:0}: Error finding container 886c90c77d8708408992548141e0292115c096e95d2cbb7bc3773d2289479947: Status 404 returned error can't find the container with id 886c90c77d8708408992548141e0292115c096e95d2cbb7bc3773d2289479947 Oct 30 09:26:22.027417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.027361 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:26:22.449731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.449695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:22.630969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.630931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:22.664347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.664318 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:26:22.694607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.694570 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz"] Oct 30 09:26:22.702577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.702487 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9371b7-mccfz"] Oct 30 09:26:22.833772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.833740 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:22.840608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.840554 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" event={"ID":"1e5bbb9b-3dc4-4795-9b56-17ae03add46c","Type":"ContainerStarted","Data":"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56"} Oct 30 09:26:22.840798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.840628 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" event={"ID":"1e5bbb9b-3dc4-4795-9b56-17ae03add46c","Type":"ContainerStarted","Data":"886c90c77d8708408992548141e0292115c096e95d2cbb7bc3773d2289479947"} Oct 30 09:26:22.840798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.840789 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:22.842546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.842516 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" event={"ID":"7bca8563-e142-4c4d-852d-f68e56057a64","Type":"ContainerStarted","Data":"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216"} Oct 30 09:26:22.842691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.842622 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:22.906302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.904147 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" podStartSLOduration=1.904125375 podStartE2EDuration="1.904125375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:22.893283573 +0000 UTC m=+1946.616710714" watchObservedRunningTime="2026-10-30 09:26:22.904125375 +0000 UTC m=+1946.627552519" Oct 30 09:26:22.933972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:22.933919 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" podStartSLOduration=1.933903751 podStartE2EDuration="1.933903751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:22.932391174 +0000 UTC m=+1946.655818317" watchObservedRunningTime="2026-10-30 09:26:22.933903751 +0000 UTC m=+1946.657330893" Oct 30 09:26:23.140549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.140503 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:23.314532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.314499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:23.420385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.420293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:23.657842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.657637 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:26:23.658044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.657903 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" containerID="cri-o://b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab" gracePeriod=30 Oct 30 09:26:23.674207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.674130 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:23.674643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:23.674625 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:26:24.105654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.105616 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:26:24.399184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.399100 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec458ac-4f29-4839-9b86-62edb2d1bf69" path="/var/lib/kubelet/pods/7ec458ac-4f29-4839-9b86-62edb2d1bf69/volumes" Oct 30 09:26:24.779295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.774670 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:26:24.834854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.818454 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:24.834854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.823007 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:26:24.834854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.833956 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c9232f-serving-cert\"" Oct 30 09:26:24.835318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.835289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:24.921006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.920969 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:24.921236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.921023 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:24.921236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.921128 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:24.921401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.921376 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4rl\" (UniqueName: \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:24.936395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:24.936359 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.025671 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4rl\" (UniqueName: \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.025817 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.025860 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.025896 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:25.026100 4920 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-serving-cert: secret "maas-api-e2e-praxis-a-c9232f-serving-cert" not found Oct 30 09:26:25.026413 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:25.026182 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls podName:ec2e7195-10fb-44e1-8333-c93b8a80e584 nodeName:}" failed. No retries permitted until 2026-10-30 09:26:25.526161971 +0000 UTC m=+1949.249589106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls") pod "maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" (UID: "ec2e7195-10fb-44e1-8333-c93b8a80e584") : secret "maas-api-e2e-praxis-a-c9232f-serving-cert" not found Oct 30 09:26:25.037025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.036987 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.040001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.039964 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.046017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.045989 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4rl\" (UniqueName: \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.288882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.288776 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:25.466928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.466890 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:25.531894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.531856 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.534410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.534387 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.705183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.705092 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:26:25.710469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.710441 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.729111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.728487 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:26:25.732744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.732709 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-20f71c-openshift-default-dockercfg-6jmmw\"" Oct 30 09:26:25.754394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.754361 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:25.762504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.762475 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836030 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836075 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836102 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48kn\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836120 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836150 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836322 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836413 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836507 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.839346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.836535 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:25.856157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.856113 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:25.966292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.965277 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.994789 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.994950 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.995081 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.995388 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.995535 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.995575 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48kn\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.000239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:25.995709 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.023102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.002392 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.023102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.018355 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.023102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.018454 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.027832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.027793 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.028625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.028596 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.029291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.029266 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.034527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.034504 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.054406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.053985 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.056627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.056596 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48kn\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.056963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.056938 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\") pod \"e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.100435 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:26.100394 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2e7195_10fb_44e1_8333_c93b8a80e584.slice/crio-9ab261f6bf96234ed0c6be38c119a68898b02b78db79266b34244c78fb381b7f WatchSource:0}: Error finding container 9ab261f6bf96234ed0c6be38c119a68898b02b78db79266b34244c78fb381b7f: Status 404 returned error can't find the container with id 9ab261f6bf96234ed0c6be38c119a68898b02b78db79266b34244c78fb381b7f Oct 30 09:26:26.100993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.100968 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:26:26.330871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.330842 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:26.588020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.587916 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-356553-openshift-default-766c847c47-482qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:26:26.588020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.587977 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:26:26.667151 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:26.667122 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36a6111_7df9_49c8_8080_ef13d03d4ab3.slice/crio-f160baa7736c2108b779d4b8c00e38da8586374f55bfd1b2c644688e6d999a6b WatchSource:0}: Error finding container f160baa7736c2108b779d4b8c00e38da8586374f55bfd1b2c644688e6d999a6b: Status 404 returned error can't find the container with id f160baa7736c2108b779d4b8c00e38da8586374f55bfd1b2c644688e6d999a6b Oct 30 09:26:26.669380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.669355 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:26:26.669623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.669591 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:26.669679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.669654 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:26.669720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.669683 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:26.962192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.961963 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" event={"ID":"ec2e7195-10fb-44e1-8333-c93b8a80e584","Type":"ContainerStarted","Data":"2da331e3367cb24771b59476e749af79df8216c2a9f5b0f90ffd51b4d9112093"} Oct 30 09:26:26.962192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.962054 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:26.963064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.962605 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" event={"ID":"ec2e7195-10fb-44e1-8333-c93b8a80e584","Type":"ContainerStarted","Data":"9ab261f6bf96234ed0c6be38c119a68898b02b78db79266b34244c78fb381b7f"} Oct 30 09:26:26.965031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.965004 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerStarted","Data":"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3"} Oct 30 09:26:26.965149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:26.965057 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerStarted","Data":"f160baa7736c2108b779d4b8c00e38da8586374f55bfd1b2c644688e6d999a6b"} Oct 30 09:26:27.004360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.004305 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" podStartSLOduration=2.6219075050000002 podStartE2EDuration="3.004289783s" totalImagesPullingTime="382.382278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:27.00083304 +0000 UTC m=+1950.724260183" watchObservedRunningTime="2026-10-30 09:26:27.004289783 +0000 UTC m=+1950.727716925" Oct 30 09:26:27.047373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.047312 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podStartSLOduration=2.047294451 podStartE2EDuration="2.047294451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:27.041491977 +0000 UTC m=+1950.764919112" watchObservedRunningTime="2026-10-30 09:26:27.047294451 +0000 UTC m=+1950.770721595" Oct 30 09:26:27.127802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.127739 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:27.331633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.331600 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:27.338405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.338378 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:27.972781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.972738 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:27.973958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:27.973935 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:26:28.870550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.870510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:28.968643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.968606 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:26:28.984022 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.983984 4920 generic.go:356] "Generic (PLEG): container finished" podID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerID="b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab" exitCode=0 Oct 30 09:26:28.984529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.984042 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" Oct 30 09:26:28.984529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.984060 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerDied","Data":"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab"} Oct 30 09:26:28.984529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.984138 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh" event={"ID":"ce3317a9-cc88-4d59-9078-7d1df022ba70","Type":"ContainerDied","Data":"26a938c5415aa2a24f9f35ed03f33fa4f8c18fa6df913211ea0509a397886601"} Oct 30 09:26:28.984529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.984165 4920 scope.go:122] "RemoveContainer" containerID="b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab" Oct 30 09:26:28.996464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:28.996440 4920 scope.go:122] "RemoveContainer" containerID="c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb" Oct 30 09:26:29.007756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.007727 4920 scope.go:122] "RemoveContainer" containerID="b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab" Oct 30 09:26:29.008134 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:29.008104 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab\": container with ID starting with b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab not found: ID does not exist" containerID="b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab" Oct 30 09:26:29.008265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.008144 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab"} err="failed to get container status \"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab\": rpc error: code = NotFound desc = could not find container \"b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab\": container with ID starting with b748c2223842fcb82f50501cb5d34ce1d2ca076543edcd52c2603e989ecc90ab not found: ID does not exist" Oct 30 09:26:29.008265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.008170 4920 scope.go:122] "RemoveContainer" containerID="c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb" Oct 30 09:26:29.008524 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:29.008502 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb\": container with ID starting with c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb not found: ID does not exist" containerID="c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb" Oct 30 09:26:29.008576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.008532 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb"} err="failed to get container status \"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb\": rpc error: code = NotFound desc = could not find container \"c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb\": container with ID starting with c9d75ee02e883aeb17461ca4bb9f6eb9c52d005e6eedfc89aed51925ccf5cdeb not found: ID does not exist" Oct 30 09:26:29.072566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072522 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.072766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072704 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.072766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072747 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.072892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072816 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.072892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072863 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.073006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072915 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.073006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.072949 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.073006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073003 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.073170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073033 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\") pod \"ce3317a9-cc88-4d59-9078-7d1df022ba70\" (UID: \"ce3317a9-cc88-4d59-9078-7d1df022ba70\") " Oct 30 09:26:29.073313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073287 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:29.073626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073601 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:29.073723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073642 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:29.073910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073889 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.073980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073915 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3317a9-cc88-4d59-9078-7d1df022ba70-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.073980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073928 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.074104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.073987 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:29.074251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.074193 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:29.075797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.075764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:29.076368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.076319 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:26:29.076368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.076323 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:29.076578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.076555 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp" pod "ce3317a9-cc88-4d59-9078-7d1df022ba70" (UID: "ce3317a9-cc88-4d59-9078-7d1df022ba70"). InnerVolumeSpecName "kube-api-access-8vcpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:29.087881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.087822 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:29.175753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175721 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.175753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175751 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.175753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175761 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.175994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175772 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vcpp\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-kube-api-access-8vcpp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.175994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175782 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.175994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.175790 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3317a9-cc88-4d59-9078-7d1df022ba70-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:29.195880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.195829 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:29.332925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.332886 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:26:29.341964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.341886 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-356553-openshift-default-766c847c47-482qh"] Oct 30 09:26:29.425607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.425575 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:26:29.425816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.425792 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" containerID="cri-o://f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a" gracePeriod=30 Oct 30 09:26:29.461939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.461907 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:29.594879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:29.594797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:30.400935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.400902 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" path="/var/lib/kubelet/pods/ce3317a9-cc88-4d59-9078-7d1df022ba70/volumes" Oct 30 09:26:30.484313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.484282 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:30.770625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.770527 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:30.828341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.828261 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:26:30.828964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.828943 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.829075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.828965 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.829075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.828985 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.829075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.828992 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.829275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.829259 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.829349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.829279 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3317a9-cc88-4d59-9078-7d1df022ba70" containerName="istio-proxy" Oct 30 09:26:30.834459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.834438 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.859602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.857148 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:26:30.861949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.861919 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9d39c0-openshift-default-dockercfg-74sfx\"" Oct 30 09:26:30.891408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.891370 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:30.907092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907055 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907174 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcc5c\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907262 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907418 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907470 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907585 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907628 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907668 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:30.907725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:30.907697 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018069 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018162 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcc5c\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018229 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018350 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018386 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018435 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018467 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018520 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.018688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018554 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.022286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.018996 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.027297 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.029368 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.033527 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.033785 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.039159 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.047514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.045627 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:31.056912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.056836 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.065743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.065268 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcc5c\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.065743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.065691 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\") pod \"e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.147347 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:31.147313 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:26:31.164718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.164691 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:31.233496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.233424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:31.338293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.338201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:31.531472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.523905 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:31.531472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.524005 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:31.531472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.524051 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:31.547446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:31.547282 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:26:32.096986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.096941 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerStarted","Data":"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa"} Oct 30 09:26:32.097170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.097011 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerStarted","Data":"56ce5db0e9aacb935d465cedd1fc3deec0db653dfcb5df310bde929f7fa5b487"} Oct 30 09:26:32.142191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.142124 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podStartSLOduration=2.142109047 podStartE2EDuration="2.142109047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:32.139001977 +0000 UTC m=+1955.862429142" watchObservedRunningTime="2026-10-30 09:26:32.142109047 +0000 UTC m=+1955.865536190" Oct 30 09:26:32.165614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.165581 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:32.170886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.170858 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:32.552125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.552091 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:32.978571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:32.978490 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:26:33.107465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:33.107436 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:33.108665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:33.108641 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:26:33.887753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:33.887726 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:26:33.888186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:33.887788 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:26:34.236598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.236513 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:26:34.241117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.241086 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.255379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.255337 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-19a563-serving-cert\"" Oct 30 09:26:34.260399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.260371 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:26:34.321597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.321549 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:34.402945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.402913 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.403123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.402972 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf5w\" (UniqueName: \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.403252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.403209 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.403323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.403306 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.421624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.421589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:34.504734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.504646 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.504734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.504700 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.504956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.504816 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.504956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.504869 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf5w\" (UniqueName: \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.505712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.505688 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.507387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.507364 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.507511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.507485 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.524508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.524474 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf5w\" (UniqueName: \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\") pod \"maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.526920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.526894 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:34.557028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.556998 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:34.648696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.648573 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:34.741098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.741062 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:26:34.751691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.751660 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:34.897656 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:34.897629 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda335af27_f817_4aca_ac59_37785b33927e.slice/crio-867ab7438db18a18c5a429d249550bab896ba4f921680410f4ecb8e3857329c6 WatchSource:0}: Error finding container 867ab7438db18a18c5a429d249550bab896ba4f921680410f4ecb8e3857329c6: Status 404 returned error can't find the container with id 867ab7438db18a18c5a429d249550bab896ba4f921680410f4ecb8e3857329c6 Oct 30 09:26:34.898902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.898878 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:26:34.913019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.912993 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913083 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913130 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913183 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913250 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913322 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913365 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913494 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913546 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\") pod \"df4c20c8-2c3a-4129-851d-2776b81c95c9\" (UID: \"df4c20c8-2c3a-4129-851d-2776b81c95c9\") " Oct 30 09:26:34.913812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.913782 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.914056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914033 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.914665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914383 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:34.914665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914394 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.914665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914429 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:34.914665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914390 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.914665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.914513 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.915593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.915568 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:34.916715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.916684 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:34.916812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.916785 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:26:34.916812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:34.916800 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq" pod "df4c20c8-2c3a-4129-851d-2776b81c95c9" (UID: "df4c20c8-2c3a-4129-851d-2776b81c95c9"). InnerVolumeSpecName "kube-api-access-ggxqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:35.015565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015525 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015558 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxqq\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-kube-api-access-ggxqq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015571 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015586 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015597 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4c20c8-2c3a-4129-851d-2776b81c95c9-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015609 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4c20c8-2c3a-4129-851d-2776b81c95c9-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.015818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.015620 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4c20c8-2c3a-4129-851d-2776b81c95c9-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:26:35.143541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.143497 4920 generic.go:356] "Generic (PLEG): container finished" podID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerID="f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a" exitCode=0 Oct 30 09:26:35.143741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.143560 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" Oct 30 09:26:35.143741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.143577 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" event={"ID":"df4c20c8-2c3a-4129-851d-2776b81c95c9","Type":"ContainerDied","Data":"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a"} Oct 30 09:26:35.143741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.143673 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j" event={"ID":"df4c20c8-2c3a-4129-851d-2776b81c95c9","Type":"ContainerDied","Data":"29e8a5a46440b2569da6f3d69e1b8dc21c60396ed14331b42c35d0b44e12da7f"} Oct 30 09:26:35.143741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.143703 4920 scope.go:122] "RemoveContainer" containerID="f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a" Oct 30 09:26:35.145140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.145108 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" event={"ID":"a335af27-f817-4aca-ac59-37785b33927e","Type":"ContainerStarted","Data":"867ab7438db18a18c5a429d249550bab896ba4f921680410f4ecb8e3857329c6"} Oct 30 09:26:35.155883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.155864 4920 scope.go:122] "RemoveContainer" containerID="f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a" Oct 30 09:26:35.156301 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:26:35.156282 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a\": container with ID starting with f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a not found: ID does not exist" containerID="f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a" Oct 30 09:26:35.156352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.156310 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a"} err="failed to get container status \"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a\": rpc error: code = NotFound desc = could not find container \"f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a\": container with ID starting with f6615191044b01b697513fd1d31de4ff7507b118b01d06ebfcba8a716fb8d30a not found: ID does not exist" Oct 30 09:26:35.198236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.198192 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:26:35.210885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:35.210848 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9371b7-openshift-default-7cc8c5668f-6fr4j"] Oct 30 09:26:36.155110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.155067 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" event={"ID":"a335af27-f817-4aca-ac59-37785b33927e","Type":"ContainerStarted","Data":"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946"} Oct 30 09:26:36.155628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.155134 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:36.197631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.197580 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" podStartSLOduration=1.777318308 podStartE2EDuration="2.197564236s" totalImagesPullingTime="420.245928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:36.19365183 +0000 UTC m=+1959.917078984" watchObservedRunningTime="2026-10-30 09:26:36.197564236 +0000 UTC m=+1959.920991379" Oct 30 09:26:36.318033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.318000 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:36.399234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.399187 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" path="/var/lib/kubelet/pods/df4c20c8-2c3a-4129-851d-2776b81c95c9/volumes" Oct 30 09:26:36.418427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:36.418346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:37.993411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:37.993381 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:39.577233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:39.577186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:40.014325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:40.014206 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:40.114920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:40.114883 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:41.054622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:41.054590 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:42.169579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:42.169548 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:26:42.637690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:42.637657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:43.537596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:43.537560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:44.288511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.288470 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:26:44.290036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.290010 4920 cpu_manager.go:414] "containerMap: removing container" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" Oct 30 09:26:44.290282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.290263 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" Oct 30 09:26:44.290755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.290734 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4c20c8-2c3a-4129-851d-2776b81c95c9" containerName="istio-proxy" Oct 30 09:26:44.302298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.301183 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.326749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.326262 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:26:44.344601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.344565 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-20f71c-serving-cert\"" Oct 30 09:26:44.359150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.359109 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.359343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.359172 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.359406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.359336 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8j5\" (UniqueName: \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.359702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.359603 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.360795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.360770 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:44.457421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.457373 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:44.460447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.460423 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8j5\" (UniqueName: \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.460856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.460834 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.460947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.460909 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.460987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.460946 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.461603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.461577 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.463541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.463519 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.463627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.463568 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.479154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.479118 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8j5\" (UniqueName: \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\") pod \"maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.567916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.566686 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:44.645663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.645626 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:44.781425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:44.781387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:45.024014 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:45.023980 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36686c13_752b_4466_a4b8_3582c26a2ae9.slice/crio-e6e79b65f13c87f0983933aac0f3cb2b8c49bcfb1964962b963a1ea5627e7db6 WatchSource:0}: Error finding container e6e79b65f13c87f0983933aac0f3cb2b8c49bcfb1964962b963a1ea5627e7db6: Status 404 returned error can't find the container with id e6e79b65f13c87f0983933aac0f3cb2b8c49bcfb1964962b963a1ea5627e7db6 Oct 30 09:26:45.026286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.026255 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:26:45.233029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.232990 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:26:45.239871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.239838 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" event={"ID":"36686c13-752b-4466-a4b8-3582c26a2ae9","Type":"ContainerStarted","Data":"e6e79b65f13c87f0983933aac0f3cb2b8c49bcfb1964962b963a1ea5627e7db6"} Oct 30 09:26:45.240109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.239965 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.261642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.261611 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d95d35-openshift-default-dockercfg-2q48b\"" Oct 30 09:26:45.268119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.268089 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:26:45.273097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273020 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273134 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273292 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273341 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273488 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273557 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273670 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273699 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.273855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.273832 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlh4b\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.297033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.296999 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380264 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380377 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380515 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380543 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380627 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380652 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380738 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlh4b\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380784 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.380813 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.381411 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.381781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.381680 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.382612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.382355 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.382685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.382609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.416240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.395816 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.416240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.403044 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.435441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.431609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.438717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.436484 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.450242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.445028 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlh4b\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\") pod \"e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.558819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.558791 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:45.835349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.832256 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:45.925707 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:26:45.925650 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda703116f_bf24_46d0_a530_533f2022386f.slice/crio-662bf8a7cf48f0068911141d3f296a86799384fd756c58b629630186a7a8b563 WatchSource:0}: Error finding container 662bf8a7cf48f0068911141d3f296a86799384fd756c58b629630186a7a8b563: Status 404 returned error can't find the container with id 662bf8a7cf48f0068911141d3f296a86799384fd756c58b629630186a7a8b563 Oct 30 09:26:45.928635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.928596 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:45.928725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.928686 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:45.928766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.928730 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:45.929072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:45.929051 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:26:46.250690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.250578 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" event={"ID":"36686c13-752b-4466-a4b8-3582c26a2ae9","Type":"ContainerStarted","Data":"2db4cf3b521a18c6e954cb3a9d8b796ee3e64dc5540a31a301c9005a6af44eeb"} Oct 30 09:26:46.250690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.250658 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:46.252562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.252529 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerStarted","Data":"16bcffef5b7c15b2eee86b164fafd7f259b88f5ebc0f24044fe754f413726c15"} Oct 30 09:26:46.252696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.252591 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerStarted","Data":"662bf8a7cf48f0068911141d3f296a86799384fd756c58b629630186a7a8b563"} Oct 30 09:26:46.330498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.330456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:46.349794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.348828 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podStartSLOduration=1.3488071910000001 podStartE2EDuration="1.348807191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:46.344971451 +0000 UTC m=+1970.068398631" watchObservedRunningTime="2026-10-30 09:26:46.348807191 +0000 UTC m=+1970.072234335" Oct 30 09:26:46.352313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.351835 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" podStartSLOduration=1.88878064 podStartE2EDuration="2.351821091s" totalImagesPullingTime="463.040451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:46.302622448 +0000 UTC m=+1970.026049592" watchObservedRunningTime="2026-10-30 09:26:46.351821091 +0000 UTC m=+1970.075248235" Oct 30 09:26:46.572291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.566236 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:46.594240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:46.589615 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:47.162281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.157488 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:47.187873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.187828 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:26:47.265047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.264806 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:47.266507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.266480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:47.267023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.267005 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:26:47.407269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.401974 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:47.424102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.424020 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd"] Oct 30 09:26:47.431074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.431047 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4fc2e2-skckd"] Oct 30 09:26:47.894482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:47.894446 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:48.052110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:48.048315 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6"] Oct 30 09:26:48.058235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:48.055028 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c94b2eb-4glg6"] Oct 30 09:26:48.400021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:48.399985 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815ccca2-b75f-42fa-967a-0fcb6536d805" path="/var/lib/kubelet/pods/815ccca2-b75f-42fa-967a-0fcb6536d805/volumes" Oct 30 09:26:48.400489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:48.400471 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9880b5e6-1079-4eed-afd7-f576879f5597" path="/var/lib/kubelet/pods/9880b5e6-1079-4eed-afd7-f576879f5597/volumes" Oct 30 09:26:48.848025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:48.847787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:49.079204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:49.079169 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:49.674725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:49.674684 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:50.011819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:50.011732 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:50.190440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:50.190406 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:50.581734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:50.581691 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:50.683515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:50.683460 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:51.103494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:51.103422 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59"] Oct 30 09:26:51.169753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:51.169712 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23df2ebe-h8f59"] Oct 30 09:26:51.562916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:51.562878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:52.270721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:52.270690 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:26:52.485620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:52.485575 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ae024a-e18b-4c67-aa6d-156bea0ca507" path="/var/lib/kubelet/pods/e9ae024a-e18b-4c67-aa6d-156bea0ca507/volumes" Oct 30 09:26:53.061340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.061273 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:26:53.070132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.070100 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.074062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.074033 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:26:53.086398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.086372 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9d39c0-serving-cert\"" Oct 30 09:26:53.170515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.170433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:53.173259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.173213 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fmb\" (UniqueName: \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.173415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.173390 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.173546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.173530 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.173610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.173588 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.260644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.260610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:53.275148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.275115 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.275601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.275249 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.275601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.275288 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.275601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.275346 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fmb\" (UniqueName: \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.275988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.275968 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.277710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.277693 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.277774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.277734 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.304245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.304193 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fmb\" (UniqueName: \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\") pod \"maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.442293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.441529 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:53.868510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.868484 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:26:53.992433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:53.992353 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:54.141295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:54.122259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:54.568249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:54.566201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:54.643255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:54.639177 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:26:54.643255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:54.639424 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" containerID="cri-o://5cc8781d67f514033fbd0e89def3297a35f3282643d633603f2d0c69ce172a38" gracePeriod=30 Oct 30 09:26:54.730793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:54.730757 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" event={"ID":"348b3649-4752-4d7e-809d-3110c58cb8cf","Type":"ContainerStarted","Data":"0d9056f18a372125d4d2fcc384fee8053ff5e5ac16e42a1aebcfb8b130fce9a9"} Oct 30 09:26:55.047277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:55.047178 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:55.745712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:55.745626 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" event={"ID":"348b3649-4752-4d7e-809d-3110c58cb8cf","Type":"ContainerStarted","Data":"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9"} Oct 30 09:26:55.746383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:55.746357 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:26:55.787114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:55.787056 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" podStartSLOduration=2.420811392 podStartE2EDuration="2.78704055s" totalImagesPullingTime="366.229158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:55.785091817 +0000 UTC m=+1979.508518974" watchObservedRunningTime="2026-10-30 09:26:55.78704055 +0000 UTC m=+1979.510467739" Oct 30 09:26:56.919795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:56.919736 4920 patch_prober.go:29] interesting pod/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:26:56.920362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:56.919801 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:26:56.986229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:56.986183 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:58.772886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.765648 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:26:58.804549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.801912 4920 generic.go:356] "Generic (PLEG): container finished" podID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerID="f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc" exitCode=0 Oct 30 09:26:58.804549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.802287 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerDied","Data":"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc"} Oct 30 09:26:58.804549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.802926 4920 scope.go:122] "RemoveContainer" containerID="f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc" Oct 30 09:26:58.806752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.806708 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:58.806863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.806801 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:58.806863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:58.806843 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:59.823241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.819310 4920 generic.go:356] "Generic (PLEG): container finished" podID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerID="3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6" exitCode=0 Oct 30 09:26:59.823241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.819460 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerDied","Data":"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6"} Oct 30 09:26:59.823241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.819991 4920 scope.go:122] "RemoveContainer" containerID="3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6" Oct 30 09:26:59.827757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.826462 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:59.827757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.826564 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:59.827757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.826611 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:26:59.830111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.830072 4920 generic.go:356] "Generic (PLEG): container finished" podID="50341379-878a-4890-b526-1ac48747bde4" containerID="5cc8781d67f514033fbd0e89def3297a35f3282643d633603f2d0c69ce172a38" exitCode=0 Oct 30 09:26:59.830242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.830114 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerDied","Data":"5cc8781d67f514033fbd0e89def3297a35f3282643d633603f2d0c69ce172a38"} Oct 30 09:26:59.830242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.830188 4920 scope.go:122] "RemoveContainer" containerID="b3a23f420c9ca12076115b427640137eb699867bd9e9ba24ceb2080383c07118" Oct 30 09:26:59.832635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.832601 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerStarted","Data":"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac"} Oct 30 09:26:59.931092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:26:59.921762 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:00.273187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.272784 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:27:00.273187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.273039 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 09:27:00.273187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.273090 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 09:27:00.445449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.443906 4920 scope.go:122] "RemoveContainer" containerID="d39be6eea88caa90ff39febdc591787ea8ef484cdd9c4a58ee1a9456d0c50e6c" Oct 30 09:27:00.450352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.450313 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:00.456525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.456497 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:27:00.460175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.459986 4920 scope.go:122] "RemoveContainer" containerID="b51aea464be017661b1344896bad835bd3eacfc52b6be3a6e99339900216d742" Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.548136 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551434 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551544 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551579 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551617 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551760 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551789 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551833 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551877 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.551962 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\") pod \"50341379-878a-4890-b526-1ac48747bde4\" (UID: \"50341379-878a-4890-b526-1ac48747bde4\") " Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.552583 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.553837 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.555462 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:00.557239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.554231 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:00.574245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.563402 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.651802 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.654050 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.654079 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50341379-878a-4890-b526-1ac48747bde4-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.654097 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50341379-878a-4890-b526-1ac48747bde4-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.654110 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.658320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.654124 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.666146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.664394 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:00.666146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.665994 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:00.668494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.668465 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d" pod "50341379-878a-4890-b526-1ac48747bde4" (UID: "50341379-878a-4890-b526-1ac48747bde4"). InnerVolumeSpecName "kube-api-access-9jz8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:00.760211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.760090 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.760211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.760129 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jz8d\" (UniqueName: \"kubernetes.io/projected/50341379-878a-4890-b526-1ac48747bde4-kube-api-access-9jz8d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.760211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.760144 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50341379-878a-4890-b526-1ac48747bde4-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:00.867279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.867247 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerStarted","Data":"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa"} Oct 30 09:27:00.869211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.869185 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" Oct 30 09:27:00.869500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.869479 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm" event={"ID":"50341379-878a-4890-b526-1ac48747bde4","Type":"ContainerDied","Data":"bbe9484040b8b175cc9c21e75e4696497adc5ce8d7f6761f102f83b4cf60656c"} Oct 30 09:27:00.869556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.869529 4920 scope.go:122] "RemoveContainer" containerID="5cc8781d67f514033fbd0e89def3297a35f3282643d633603f2d0c69ce172a38" Oct 30 09:27:00.969237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.967602 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:27:00.972144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:00.972109 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4fc2e2-openshift-default-8df6c4b49-2xtvm"] Oct 30 09:27:01.064386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.064134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:01.176073 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:27:01.175984 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:27:01.273199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.273162 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 09:27:01.273447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.273245 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 09:27:01.473520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.473428 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:27:01.474516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.474487 4920 cpu_manager.go:414] "containerMap: removing container" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:01.474644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.474517 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:01.474767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.474751 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:01.474826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.474776 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:01.484516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.484489 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.535240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.525354 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d3de9d-openshift-default-dockercfg-6cq7q\"" Oct 30 09:27:01.535240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.525850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:01.574250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.573789 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:27:01.628858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.628732 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672132 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672199 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5qv\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672285 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672317 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672371 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672452 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672503 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672542 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.678266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.672637 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.696234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.696187 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:27:01.714244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.696638 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 09:27:01.716343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.716305 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 09:27:01.778586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.778475 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.783345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.783305 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5qv\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.783623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.783605 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.783746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.783732 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.783885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.783871 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.784049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.784034 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.784172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.784159 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.784305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.784290 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.784505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.784492 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.785057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.785035 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.800239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.785816 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.802020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.787328 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.802477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.787556 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.802657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.787733 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.812317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.810962 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:27:01.838309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.838194 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5qv\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.859743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.858851 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.881744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.881714 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:01.966255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:01.952723 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:02.154061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.152607 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:02.179024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.178998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:02.280240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.280123 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 09:27:02.280240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.280194 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 09:27:02.428298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.417021 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50341379-878a-4890-b526-1ac48747bde4" path="/var/lib/kubelet/pods/50341379-878a-4890-b526-1ac48747bde4/volumes" Oct 30 09:27:02.562535 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:27:02.562491 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3757924f_64e9_4d9b_a37c_45fe23eb2637.slice/crio-040ecf75ddf4720ae2d9d7568c3904f8afef47d7676612cb92ea645e7c88fb8b WatchSource:0}: Error finding container 040ecf75ddf4720ae2d9d7568c3904f8afef47d7676612cb92ea645e7c88fb8b: Status 404 returned error can't find the container with id 040ecf75ddf4720ae2d9d7568c3904f8afef47d7676612cb92ea645e7c88fb8b Oct 30 09:27:02.563332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.563309 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:27:02.565277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.565244 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:02.565350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.565310 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:02.565350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.565341 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:02.689929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.689901 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 09:27:02.690066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.689955 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 09:27:02.771358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:02.771232 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:27:03.053239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.048284 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerStarted","Data":"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e"} Oct 30 09:27:03.053239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.048347 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerStarted","Data":"040ecf75ddf4720ae2d9d7568c3904f8afef47d7676612cb92ea645e7c88fb8b"} Oct 30 09:27:03.101243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.097549 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podStartSLOduration=2.0975295369999998 podStartE2EDuration="2.097529537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:03.095506177 +0000 UTC m=+1986.818933349" watchObservedRunningTime="2026-10-30 09:27:03.097529537 +0000 UTC m=+1986.820956682" Oct 30 09:27:03.158297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.154582 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:03.160568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.160528 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:03.160739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.160590 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:03.279242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.278815 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 09:27:03.279242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.278873 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 09:27:03.689686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.689655 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 09:27:03.689841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.689703 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 09:27:03.837240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:03.836817 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:04.159549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:04.159517 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:04.273399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:04.273362 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 09:27:04.273554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:04.273409 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 09:27:04.689909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:04.689869 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 09:27:04.690101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:04.689920 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 09:27:05.071161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.071126 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:05.072295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.072274 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:05.279635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.279595 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:27:05.280143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.280120 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:27:05.280813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.280792 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:27:05.537726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.537682 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:27:05.576307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.576273 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:27:05.628244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.626369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:05.690324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.690280 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 09:27:05.690482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.690342 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 09:27:05.767912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.767873 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:05.883713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:05.883622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:06.697683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:06.697646 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:27:06.699690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:06.699662 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:27:07.294295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:07.294260 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:07.392816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:07.392783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:07.675427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:07.666891 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:09.426881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:09.426850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:09.527318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:09.527280 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:09.627116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:09.627082 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:10.097042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.097004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:10.538524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.538477 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" podUID="20e13869-457b-4d01-a351-fc1fe72c920b" containerName="payload-processing" containerID="cri-o://8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055" gracePeriod=25 Oct 30 09:27:10.576617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.576575 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" podUID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" containerName="payload-pre-processing" containerID="cri-o://ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3" gracePeriod=25 Oct 30 09:27:10.878433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.878397 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:27:10.954862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.954831 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:27:10.996409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996375 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\") pod \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " Oct 30 09:27:10.996610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996454 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\") pod \"20e13869-457b-4d01-a351-fc1fe72c920b\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " Oct 30 09:27:10.996676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996634 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\" (UniqueName: \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\") pod \"20e13869-457b-4d01-a351-fc1fe72c920b\" (UID: \"20e13869-457b-4d01-a351-fc1fe72c920b\") " Oct 30 09:27:10.996736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996676 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\" (UniqueName: \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\") pod \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\" (UID: \"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22\") " Oct 30 09:27:10.997269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996840 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume" pod "5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" (UID: "5ed3b183-6d23-4a6b-96e1-7fdafb77fb22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:10.997269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.996869 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume" pod "20e13869-457b-4d01-a351-fc1fe72c920b" (UID: "20e13869-457b-4d01-a351-fc1fe72c920b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:10.997541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.997511 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:10.997598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:10.997548 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e13869-457b-4d01-a351-fc1fe72c920b-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:11.032454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.002604 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2" pod "20e13869-457b-4d01-a351-fc1fe72c920b" (UID: "20e13869-457b-4d01-a351-fc1fe72c920b"). InnerVolumeSpecName "kube-api-access-58ln2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:11.038398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.038357 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46" pod "5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" (UID: "5ed3b183-6d23-4a6b-96e1-7fdafb77fb22"). InnerVolumeSpecName "kube-api-access-wrs46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:11.077538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.077497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:11.099188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.099088 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ln2\" (UniqueName: \"kubernetes.io/projected/20e13869-457b-4d01-a351-fc1fe72c920b-kube-api-access-58ln2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:11.099188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.099129 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrs46\" (UniqueName: \"kubernetes.io/projected/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22-kube-api-access-wrs46\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:11.148244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.142911 4920 generic.go:356] "Generic (PLEG): container finished" podID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" containerID="ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3" exitCode=0 Oct 30 09:27:11.148244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.143072 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" event={"ID":"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22","Type":"ContainerDied","Data":"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3"} Oct 30 09:27:11.148244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.143124 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" event={"ID":"5ed3b183-6d23-4a6b-96e1-7fdafb77fb22","Type":"ContainerDied","Data":"b36a383ff45c505c3b8fd315624a3284b493bf25cc2b8e205cf77182e649bd11"} Oct 30 09:27:11.148244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.143152 4920 scope.go:122] "RemoveContainer" containerID="ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3" Oct 30 09:27:11.148244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.143475 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll" Oct 30 09:27:11.154573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.154433 4920 generic.go:356] "Generic (PLEG): container finished" podID="20e13869-457b-4d01-a351-fc1fe72c920b" containerID="8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055" exitCode=0 Oct 30 09:27:11.154573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.154503 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" Oct 30 09:27:11.154573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.154534 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" event={"ID":"20e13869-457b-4d01-a351-fc1fe72c920b","Type":"ContainerDied","Data":"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055"} Oct 30 09:27:11.154861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.154659 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj" event={"ID":"20e13869-457b-4d01-a351-fc1fe72c920b","Type":"ContainerDied","Data":"c9a5b7a5785c57dbed4bc0f09ba540c7135932a3240c05b7400d8e9cdeb3e81e"} Oct 30 09:27:11.166811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.166789 4920 scope.go:122] "RemoveContainer" containerID="ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3" Oct 30 09:27:11.167142 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:27:11.167122 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3\": container with ID starting with ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3 not found: ID does not exist" containerID="ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3" Oct 30 09:27:11.167266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.167157 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3"} err="failed to get container status \"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3\": rpc error: code = NotFound desc = could not find container \"ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3\": container with ID starting with ed4f29cce9156836e660944e6a075244bd352dc70ce838f9059117a2707a0cd3 not found: ID does not exist" Oct 30 09:27:11.167266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.167181 4920 scope.go:122] "RemoveContainer" containerID="8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055" Oct 30 09:27:11.179343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.179315 4920 scope.go:122] "RemoveContainer" containerID="8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055" Oct 30 09:27:11.179632 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:27:11.179611 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055\": container with ID starting with 8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055 not found: ID does not exist" containerID="8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055" Oct 30 09:27:11.179713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.179640 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055"} err="failed to get container status \"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055\": rpc error: code = NotFound desc = could not find container \"8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055\": container with ID starting with 8ba7b6cd716fff6bb9dc60838edfd6a576c99d94857e4be89399996315885055 not found: ID does not exist" Oct 30 09:27:11.216538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.216500 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:27:11.220940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.220905 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5491bf-595b9bf4f9-jdkll"] Oct 30 09:27:11.255008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.254974 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:27:11.258439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.258417 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5491bf-769f566db6-zq5jj"] Oct 30 09:27:11.366452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.366377 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:11.834108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.834076 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:27:11.835026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835004 4920 cpu_manager.go:414] "containerMap: removing container" podUID="20e13869-457b-4d01-a351-fc1fe72c920b" containerName="payload-processing" Oct 30 09:27:11.835026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835026 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e13869-457b-4d01-a351-fc1fe72c920b" containerName="payload-processing" Oct 30 09:27:11.835257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835090 4920 cpu_manager.go:414] "containerMap: removing container" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:11.835257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835099 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50341379-878a-4890-b526-1ac48747bde4" containerName="istio-proxy" Oct 30 09:27:11.835257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835110 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" containerName="payload-pre-processing" Oct 30 09:27:11.835257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835118 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" containerName="payload-pre-processing" Oct 30 09:27:11.835464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835308 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e13869-457b-4d01-a351-fc1fe72c920b" containerName="payload-processing" Oct 30 09:27:11.835464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.835338 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" containerName="payload-pre-processing" Oct 30 09:27:11.851274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.840544 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.860417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.860340 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cbd96e-openshift-default-dockercfg-mfttj\"" Oct 30 09:27:11.876116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.876065 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:27:11.887578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.887543 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:11.910185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910149 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910294 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910378 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910438 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910465 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910523 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2wq\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910602 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910638 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.910696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.910669 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:11.994716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:11.994683 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:12.011625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.011589 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.011815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.011761 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.011815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.011808 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.011929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.011878 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2wq\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.011992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.011969 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012001 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012008 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012055 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012092 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012304 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012494 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012596 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.012878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.012859 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.013162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.013141 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.014735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.014702 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.014944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.014927 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.033265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.033137 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2wq\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.045433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.034471 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\") pod \"e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.176337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.166464 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:12.400317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.400279 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e13869-457b-4d01-a351-fc1fe72c920b" path="/var/lib/kubelet/pods/20e13869-457b-4d01-a351-fc1fe72c920b/volumes" Oct 30 09:27:12.400759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.400743 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed3b183-6d23-4a6b-96e1-7fdafb77fb22" path="/var/lib/kubelet/pods/5ed3b183-6d23-4a6b-96e1-7fdafb77fb22/volumes" Oct 30 09:27:12.563480 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:27:12.563448 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f740194_fd61_4e36_b6b5_e83de6aafa9d.slice/crio-21196b24ba790390431f0d93fc5d6b5b60e0a830398fd0a7eab4b9abeb5cfbce WatchSource:0}: Error finding container 21196b24ba790390431f0d93fc5d6b5b60e0a830398fd0a7eab4b9abeb5cfbce: Status 404 returned error can't find the container with id 21196b24ba790390431f0d93fc5d6b5b60e0a830398fd0a7eab4b9abeb5cfbce Oct 30 09:27:12.565279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.565252 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:27:12.566202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.566167 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:12.566334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.566264 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:12.566334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.566295 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:12.739690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:12.739656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:13.206979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:13.206938 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerStarted","Data":"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4"} Oct 30 09:27:13.207493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:13.207007 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerStarted","Data":"21196b24ba790390431f0d93fc5d6b5b60e0a830398fd0a7eab4b9abeb5cfbce"} Oct 30 09:27:13.264062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:13.264009 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podStartSLOduration=2.263996045 podStartE2EDuration="2.263996045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:13.261180684 +0000 UTC m=+1996.984607830" watchObservedRunningTime="2026-10-30 09:27:13.263996045 +0000 UTC m=+1996.987423187" Oct 30 09:27:14.171483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.171447 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:14.177432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.177403 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:14.217248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.217202 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:14.218198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.218177 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:27:14.468947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.468865 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:14.511396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:14.511361 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:27:15.749784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:15.749746 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:15.857604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:15.857564 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:16.745241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:16.744718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:18.343238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:18.342413 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:18.463310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:18.463237 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:18.598455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:18.598341 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:20.084628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:20.084583 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:20.712240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:20.711698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:21.274964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:21.274924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:21.375319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:21.375287 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:21.592660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:21.592609 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:21.776109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:21.776066 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:22.041774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.041725 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:27:22.055235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.053444 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.071045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.069027 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:27:22.074458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.074434 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-51e091-openshift-default-dockercfg-fjg7p\"" Oct 30 09:27:22.090574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.090522 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:22.163074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163039 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163078 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163103 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163184 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163235 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163353 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163393 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfb9\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163501 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.163549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.163532 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.185649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.185613 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.264862 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.264930 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265065 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265091 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265125 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265156 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265192 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265287 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.265342 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfb9\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.266574 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.267240 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.267485 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.267703 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.269026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.267920 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.276963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.274995 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.333318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.333181 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.357357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.357258 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfb9\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.385429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.360431 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\") pod \"e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.386779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.386485 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:22.755850 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:27:22.755811 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd72d13_1e47_400e_a718_838226d97007.slice/crio-07bda4cd6c17aa31911212e80d91b6ed09885ef27e5b28200532ee2170e8e8cc WatchSource:0}: Error finding container 07bda4cd6c17aa31911212e80d91b6ed09885ef27e5b28200532ee2170e8e8cc: Status 404 returned error can't find the container with id 07bda4cd6c17aa31911212e80d91b6ed09885ef27e5b28200532ee2170e8e8cc Oct 30 09:27:22.757913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.757884 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:27:22.759737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.759697 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:22.759852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.759784 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:22.759852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:22.759828 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:23.408237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:23.402068 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerStarted","Data":"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0"} Oct 30 09:27:23.408237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:23.402126 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerStarted","Data":"07bda4cd6c17aa31911212e80d91b6ed09885ef27e5b28200532ee2170e8e8cc"} Oct 30 09:27:23.550692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:23.550651 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:24.387305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:24.387260 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:24.414787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:24.414755 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:24.441969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:24.441935 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:24.446480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:24.446442 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:27:24.487376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:24.487322 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podStartSLOduration=2.487307688 podStartE2EDuration="2.487307688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:23.470925139 +0000 UTC m=+2007.194352282" watchObservedRunningTime="2026-10-30 09:27:24.487307688 +0000 UTC m=+2008.210734830" Oct 30 09:27:25.332311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:25.332278 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:25.435881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:25.435734 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:25.534875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:25.534844 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:25.916856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:25.916013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:26.057811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.057778 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:26.468675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.468644 4920 generic.go:356] "Generic (PLEG): container finished" podID="bf103a26-1163-4f3d-8d4a-09481a716766" containerID="263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa" exitCode=0 Oct 30 09:27:26.469205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.468721 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerDied","Data":"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa"} Oct 30 09:27:26.470622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.470595 4920 scope.go:122] "RemoveContainer" containerID="263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa" Oct 30 09:27:26.470762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.470709 4920 generic.go:356] "Generic (PLEG): container finished" podID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerID="00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3" exitCode=0 Oct 30 09:27:26.470835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.470780 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerDied","Data":"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3"} Oct 30 09:27:26.471611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.471322 4920 scope.go:122] "RemoveContainer" containerID="00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3" Oct 30 09:27:26.472354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472328 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:26.472457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472410 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:26.472521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472460 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:26.472772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472743 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:26.472842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472815 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:26.472897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:26.472861 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:27.481488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:27.481445 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerStarted","Data":"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c"} Oct 30 09:27:27.484433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:27.484406 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerStarted","Data":"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120"} Oct 30 09:27:27.493632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:27.493609 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:27.973410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:27.973368 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:27:28.165093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.165054 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:27:28.166144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.166118 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 09:27:28.166224 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.166172 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 09:27:28.335243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.334903 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:27:28.340441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.340404 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:27:28.342134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.342097 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 09:27:28.472019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.471978 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:28.569817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:28.569776 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:29.165367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:29.165327 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 09:27:29.165566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:29.165386 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 09:27:29.263229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:29.263176 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:29.338685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:29.338646 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:27:29.339016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:29.338986 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 09:27:30.165985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:30.165945 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 09:27:30.166413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:30.166008 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 09:27:30.352242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:30.351850 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:27:30.352242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:30.351912 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 09:27:30.920812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:30.920774 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:31.165884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:31.165857 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 09:27:31.166998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:31.166968 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 09:27:31.189083 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:27:31.188508 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:27:31.349241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:31.348257 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:27:31.349241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:31.348323 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 09:27:32.165102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:32.165064 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 09:27:32.165332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:32.165126 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 09:27:32.339349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:32.339308 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:27:32.340080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:32.340048 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Oct 30 09:27:32.766703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:32.766642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:33.107874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:33.107831 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:27:33.301192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:33.301157 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:27:33.337437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:33.337407 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:27:33.338432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:33.338412 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:27:33.687868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:33.687840 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:27:34.462231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:34.462192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:34.628149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:34.628116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:36.251822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:36.251786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:36.355789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:36.355760 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:37.913181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:37.913145 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:39.175452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:39.175377 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:39.499972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:39.499835 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:40.623702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.623657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:40.725730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.725694 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:40.740039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.740001 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:27:40.745865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.745835 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.761532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.761502 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c9232f\"/\"kube-root-ca.crt\"" Oct 30 09:27:40.769687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.769646 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c9232f\"/\"openshift-service-ca.crt\"" Oct 30 09:27:40.771333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.771305 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:27:40.773568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.773538 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c9232f\"/\"default-dockercfg-s8xxn\"" Oct 30 09:27:40.775309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.774108 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c9232f\"/\"praxis-route-c9232f-kserve-self-signed-certs\"" Oct 30 09:27:40.832315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.832265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:40.849897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.849861 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.850111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.849912 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.850111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.850026 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjrw\" (UniqueName: \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.850238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.850114 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.850411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.850379 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.850511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.850490 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.968483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.964501 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.968767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.966682 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.969043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.969017 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.969195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.969182 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.969892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.969372 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.970395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.970359 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjrw\" (UniqueName: \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.970980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.970956 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.973515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.973491 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.974138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.974052 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:40.999237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:40.988339 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:41.000353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.000323 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:41.039239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.036036 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjrw\" (UniqueName: \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\") pod \"praxis-route-c9232f-kserve-8db8bf5c9-z48tf\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:41.065245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.064496 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:41.474238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.473393 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:27:41.867066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.865852 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" event={"ID":"4dfa1636-fa00-4892-82a9-2cbfdb3b8680","Type":"ContainerStarted","Data":"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86"} Oct 30 09:27:41.867066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:41.865911 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" event={"ID":"4dfa1636-fa00-4892-82a9-2cbfdb3b8680","Type":"ContainerStarted","Data":"59446b2b5de5fe48e7cfba7baa8eb8b83d6742798cbe0396238b9c9933024ecb"} Oct 30 09:27:42.077243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:42.069700 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:42.317822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:42.317681 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:44.179061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:44.179009 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:44.486277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:44.486176 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:45.773281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:45.773244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:47.042932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.042876 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" podStartSLOduration=7.042861059 podStartE2EDuration="7.042861059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:41.934033441 +0000 UTC m=+2025.657460588" watchObservedRunningTime="2026-10-30 09:27:47.042861059 +0000 UTC m=+2030.766288201" Oct 30 09:27:47.043684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.043656 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:27:47.044055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.044029 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" podUID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" containerName="maas-api" containerID="cri-o://190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1" gracePeriod=30 Oct 30 09:27:47.111035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.110809 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:47.585975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.585942 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:27:47.707058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.706964 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\") pod \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " Oct 30 09:27:47.707058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.707037 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\") pod \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " Oct 30 09:27:47.707367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.707069 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\") pod \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " Oct 30 09:27:47.707367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.707280 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\" (UniqueName: \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\") pod \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\" (UID: \"8d4dd8b8-0a42-461e-90f2-e122b193bc7c\") " Oct 30 09:27:47.708626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.708597 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca" pod "8d4dd8b8-0a42-461e-90f2-e122b193bc7c" (UID: "8d4dd8b8-0a42-461e-90f2-e122b193bc7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:47.712717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.712689 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs" pod "8d4dd8b8-0a42-461e-90f2-e122b193bc7c" (UID: "8d4dd8b8-0a42-461e-90f2-e122b193bc7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:27:47.713629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.713588 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf" pod "8d4dd8b8-0a42-461e-90f2-e122b193bc7c" (UID: "8d4dd8b8-0a42-461e-90f2-e122b193bc7c"). InnerVolumeSpecName "kube-api-access-74qsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:47.714260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.714209 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls" pod "8d4dd8b8-0a42-461e-90f2-e122b193bc7c" (UID: "8d4dd8b8-0a42-461e-90f2-e122b193bc7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:27:47.747587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.747269 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:47.810459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.810361 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:47.810459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.810400 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:47.810459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.810415 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:47.810459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.810430 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qsf\" (UniqueName: \"kubernetes.io/projected/8d4dd8b8-0a42-461e-90f2-e122b193bc7c-kube-api-access-74qsf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:27:47.843254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.842650 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:47.949237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.946069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:47.985790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.985710 4920 generic.go:356] "Generic (PLEG): container finished" podID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" containerID="190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1" exitCode=0 Oct 30 09:27:47.985790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.985767 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" Oct 30 09:27:47.985997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.985790 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" event={"ID":"8d4dd8b8-0a42-461e-90f2-e122b193bc7c","Type":"ContainerDied","Data":"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1"} Oct 30 09:27:47.985997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.985858 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk" event={"ID":"8d4dd8b8-0a42-461e-90f2-e122b193bc7c","Type":"ContainerDied","Data":"5167d39f3d3922dae7946b688ed1e6ddf0c28701aff12a80a527775c58b22079"} Oct 30 09:27:47.985997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:47.985892 4920 scope.go:122] "RemoveContainer" containerID="190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1" Oct 30 09:27:48.012881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.012020 4920 scope.go:122] "RemoveContainer" containerID="190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1" Oct 30 09:27:48.012881 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:27:48.012349 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1\": container with ID starting with 190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1 not found: ID does not exist" containerID="190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1" Oct 30 09:27:48.012881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.012389 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1"} err="failed to get container status \"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1\": rpc error: code = NotFound desc = could not find container \"190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1\": container with ID starting with 190f6e85af1b620f25968ed7357b9aa768557024705061416a5e3ee7a6927dd1 not found: ID does not exist" Oct 30 09:27:48.057121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.057089 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:27:48.061556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.061525 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5491bf-59c99d6b88-r9cxk"] Oct 30 09:27:48.362038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.361997 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:48.402402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.402355 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" path="/var/lib/kubelet/pods/8d4dd8b8-0a42-461e-90f2-e122b193bc7c/volumes" Oct 30 09:27:48.404907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.404859 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:27:48.405060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:48.404969 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:27:49.756343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:49.756306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:49.866157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:49.866093 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:50.036427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:50.036394 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:50.198924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:50.198887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:50.498300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:50.496539 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": read tcp 10.132.0.2:59494->10.132.0.186:15021: read: connection reset by peer" start-of-body= Oct 30 09:27:50.498300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:50.496660 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": read tcp 10.132.0.2:59494->10.132.0.186:15021: read: connection reset by peer" Oct 30 09:27:51.057646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.057607 4920 generic.go:356] "Generic (PLEG): container finished" podID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerID="671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e" exitCode=0 Oct 30 09:27:51.058083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.057677 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerDied","Data":"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e"} Oct 30 09:27:51.058129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.058112 4920 scope.go:122] "RemoveContainer" containerID="671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e" Oct 30 09:27:51.059932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.059895 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:51.060053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.059986 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:51.060126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.060068 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:51.065197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.065166 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:51.078824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.078785 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:51.685362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:51.685327 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:52.068497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.068465 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerStarted","Data":"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771"} Oct 30 09:27:52.070323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.070297 4920 generic.go:356] "Generic (PLEG): container finished" podID="a703116f-bf24-46d0-a530-533f2022386f" containerID="16bcffef5b7c15b2eee86b164fafd7f259b88f5ebc0f24044fe754f413726c15" exitCode=0 Oct 30 09:27:52.070463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.070378 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerDied","Data":"16bcffef5b7c15b2eee86b164fafd7f259b88f5ebc0f24044fe754f413726c15"} Oct 30 09:27:52.070821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.070802 4920 scope.go:122] "RemoveContainer" containerID="16bcffef5b7c15b2eee86b164fafd7f259b88f5ebc0f24044fe754f413726c15" Oct 30 09:27:52.072642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.072607 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:52.072757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.072699 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:52.072815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.072753 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:52.079541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.079516 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:27:52.153367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.153341 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:52.153593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.153570 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:52.153650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:52.153620 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:53.081296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.081259 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerStarted","Data":"58d366b7d38f6081ac205cf8fbc3be66faa0def689e13aad6c49f95651e5df5e"} Oct 30 09:27:53.157241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.156662 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:53.157241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.156740 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:53.559847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.559807 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:27:53.560026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.560000 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:53.560077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.560048 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:53.718244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:53.707016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:54.153804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.153758 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:54.154330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.153824 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:54.559201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.559165 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:54.559401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.559238 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:54.715372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.715334 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:54.827919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:54.827789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:55.073901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.073867 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:27:55.074391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.074360 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" containerID="cri-o://615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105" gracePeriod=30 Oct 30 09:27:55.080068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.079940 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:55.168270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.157355 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:55.168270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.157415 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:55.559444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.559404 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:55.559664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.559456 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:55.667812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:55.667744 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:56.055525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.055443 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:56.156251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.155754 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:56.156251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.155816 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:56.440279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.440170 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:56.488682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.488649 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:27:56.489767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.489740 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" containerName="maas-api" Oct 30 09:27:56.489767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.489763 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" containerName="maas-api" Oct 30 09:27:56.489966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.489922 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4dd8b8-0a42-461e-90f2-e122b193bc7c" containerName="maas-api" Oct 30 09:27:56.494861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.494834 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.530030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.529835 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d87c85-openshift-default-dockercfg-cjd75\"" Oct 30 09:27:56.567678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.563318 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:56.567678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.563377 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:56.568736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.568562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:56.571155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.571112 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:27:56.671142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671098 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671261 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671298 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671367 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671467 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671606 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671660 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqp5\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671708 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.671759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.671742 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.794302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.778454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:56.796392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.795752 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.796668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.796641 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.796837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.796820 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.838076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.838040 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.839380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.839356 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.840389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.840368 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqp5\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.840888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.840865 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.841380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.841364 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.851716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.851684 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.853253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.797706 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.853425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.841290 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.853544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.839813 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.853638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.838775 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.853731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.852674 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.854442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.854420 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.867447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.867417 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.870906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.870866 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:56.895972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:56.895930 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqp5\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\") pod \"e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:57.154290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.153749 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 09:27:57.154290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.153812 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 09:27:57.160036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.160014 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d87c85-openshift-default-dockercfg-cjd75\"" Oct 30 09:27:57.166819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.166760 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:57.248655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.248584 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:57.560050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.560013 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:57.560453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.560075 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:57.588946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.588918 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:27:57.589093 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:27:57.588928 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8adee7_a31f_4b9b_8bf7_e8604e434528.slice/crio-94d280f1591e6fe887e48fde04026abb95cea10516d4c50f07f30141ec93c72a WatchSource:0}: Error finding container 94d280f1591e6fe887e48fde04026abb95cea10516d4c50f07f30141ec93c72a: Status 404 returned error can't find the container with id 94d280f1591e6fe887e48fde04026abb95cea10516d4c50f07f30141ec93c72a Oct 30 09:27:57.592303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.592263 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:57.592409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.592341 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:57.592464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:57.592410 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:58.158272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.157673 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:58.158272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.158163 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:58.158996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.158977 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:27:58.185663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.185626 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerStarted","Data":"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700"} Oct 30 09:27:58.185844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.185689 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerStarted","Data":"94d280f1591e6fe887e48fde04026abb95cea10516d4c50f07f30141ec93c72a"} Oct 30 09:27:58.251202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.251134 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podStartSLOduration=2.251113607 podStartE2EDuration="2.251113607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:58.24706911 +0000 UTC m=+2041.970496279" watchObservedRunningTime="2026-10-30 09:27:58.251113607 +0000 UTC m=+2041.974540752" Oct 30 09:27:58.559545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.559504 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 09:27:58.559765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.559564 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 09:27:58.961344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:58.961241 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:27:59.167995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.167954 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:59.173979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.173950 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:59.194561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.194530 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:59.195445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.195423 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:27:59.564450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.564415 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:27:59.564728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.564709 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:27:59.565594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:27:59.565577 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:28:00.383408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.383383 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:28:00.515118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515082 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515159 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515316 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515351 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515389 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515459 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515547 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515582 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515643 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\") pod \"cfebb963-2069-47e8-a15c-c6515b0af8a2\" (UID: \"cfebb963-2069-47e8-a15c-c6515b0af8a2\") " Oct 30 09:28:00.515799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.515724 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:00.516120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516094 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:00.516592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516122 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:00.516592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516266 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:00.516592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516462 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.516592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516515 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.516592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.516571 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:00.517701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.517665 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:00.517952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.517924 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:00.518060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.518035 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "kube-api-access-vcbwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:00.518405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.518387 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo" pod "cfebb963-2069-47e8-a15c-c6515b0af8a2" (UID: "cfebb963-2069-47e8-a15c-c6515b0af8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:28:00.618118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618031 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcbwn\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-kube-api-access-vcbwn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618072 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618086 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfebb963-2069-47e8-a15c-c6515b0af8a2-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618100 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618114 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618127 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.618442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.618140 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfebb963-2069-47e8-a15c-c6515b0af8a2-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:00.713617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.713583 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:00.717674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.717552 4920 scope.go:122] "RemoveContainer" containerID="e45e11408b91265e3f91c8bb93be68b70665d13a774b55d1d773268138e1b1bb" Oct 30 09:28:00.732614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.732476 4920 scope.go:122] "RemoveContainer" containerID="b47e6bd5ead5c68d822f9dedd37ce93c42a738587f14308322534bd91315f25b" Oct 30 09:28:00.746176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.746150 4920 scope.go:122] "RemoveContainer" containerID="bd880bfe09362facfb088783e5dc679ea535b79cc5efffaedd433c9cd7578f43" Oct 30 09:28:00.756943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.756916 4920 scope.go:122] "RemoveContainer" containerID="a09c96736e89956855cb9c9640a67c201471d00e02b935fc0fb9e831fac6ec83" Oct 30 09:28:00.766822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.766803 4920 scope.go:122] "RemoveContainer" containerID="c45cce240c5095af8963f0ea24ccafe53db453c10df6924516b384178f3aba27" Oct 30 09:28:00.776226 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.776194 4920 scope.go:122] "RemoveContainer" containerID="a3826878c11291583bb28a0a240b48d2ce6575ec398e22e67d5024243dfd488a" Oct 30 09:28:00.822342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.822295 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:00.922334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:00.922205 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:01.145460 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:01.145422 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:28:01.219296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.218908 4920 generic.go:356] "Generic (PLEG): container finished" podID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerID="615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105" exitCode=0 Oct 30 09:28:01.219296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.218941 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerDied","Data":"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105"} Oct 30 09:28:01.219296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.218998 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" Oct 30 09:28:01.219296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.219001 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb" event={"ID":"cfebb963-2069-47e8-a15c-c6515b0af8a2","Type":"ContainerDied","Data":"8d7790a0d4890face837029631b616f43dc31ea795110fea2e70d1da8506c19c"} Oct 30 09:28:01.219296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.219014 4920 scope.go:122] "RemoveContainer" containerID="615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105" Oct 30 09:28:01.251135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.251104 4920 scope.go:122] "RemoveContainer" containerID="a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8" Oct 30 09:28:01.261391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.261374 4920 scope.go:122] "RemoveContainer" containerID="615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105" Oct 30 09:28:01.261685 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:01.261666 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105\": container with ID starting with 615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105 not found: ID does not exist" containerID="615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105" Oct 30 09:28:01.261736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.261694 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105"} err="failed to get container status \"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105\": rpc error: code = NotFound desc = could not find container \"615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105\": container with ID starting with 615d128182c135e187b44ca562f26b4a3b34dbc744efe9bef2ee4396a3296105 not found: ID does not exist" Oct 30 09:28:01.261736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.261710 4920 scope.go:122] "RemoveContainer" containerID="a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8" Oct 30 09:28:01.261937 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:01.261921 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8\": container with ID starting with a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8 not found: ID does not exist" containerID="a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8" Oct 30 09:28:01.261977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.261942 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8"} err="failed to get container status \"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8\": rpc error: code = NotFound desc = could not find container \"a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8\": container with ID starting with a5934c268b9f7c0108b0af5be26502b54f5482a0b28260be69dd06827ce920f8 not found: ID does not exist" Oct 30 09:28:01.290552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.290520 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:28:01.303930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:01.303896 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5491bf-openshift-default-7fd7896757-j25wb"] Oct 30 09:28:02.402843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:02.402806 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" path="/var/lib/kubelet/pods/cfebb963-2069-47e8-a15c-c6515b0af8a2/volumes" Oct 30 09:28:02.426262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:02.426204 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:02.522142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:02.522103 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:02.879881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:02.879233 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:04.712368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:04.712331 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:06.605552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:06.605508 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:06.961879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:06.954174 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:07.402238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:07.397385 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:28:08.846201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:08.846156 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:28:08.887568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:08.887529 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:28:08.891026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:08.890996 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:08.991701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:08.991652 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:09.120124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:09.120034 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:10.651866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:10.651830 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:12.236242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:12.231947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:13.735242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:13.728104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:13.855344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:13.855293 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerName="payload-processing" containerID="cri-o://882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56" gracePeriod=25 Oct 30 09:28:13.885739 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:28:13.885466 4920 logging.go:55] [core] [Channel #1377 SubChannel #1379]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: connect: connection refused" Oct 30 09:28:13.888739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:13.888707 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" podUID="7bca8563-e142-4c4d-852d-f68e56057a64" containerName="payload-pre-processing" containerID="cri-o://f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216" gracePeriod=25 Oct 30 09:28:14.362742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.362509 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:28:14.469657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.469627 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:28:14.488258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.488204 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\") pod \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " Oct 30 09:28:14.488436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.488339 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\" (UniqueName: \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\") pod \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\" (UID: \"1e5bbb9b-3dc4-4795-9b56-17ae03add46c\") " Oct 30 09:28:14.488637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.488578 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume" pod "1e5bbb9b-3dc4-4795-9b56-17ae03add46c" (UID: "1e5bbb9b-3dc4-4795-9b56-17ae03add46c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:14.489063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.489041 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:14.490048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490024 4920 generic.go:356] "Generic (PLEG): container finished" podID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerID="882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56" exitCode=0 Oct 30 09:28:14.490145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490060 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" Oct 30 09:28:14.490145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490108 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" event={"ID":"1e5bbb9b-3dc4-4795-9b56-17ae03add46c","Type":"ContainerDied","Data":"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56"} Oct 30 09:28:14.490329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490169 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" event={"ID":"1e5bbb9b-3dc4-4795-9b56-17ae03add46c","Type":"ContainerDied","Data":"886c90c77d8708408992548141e0292115c096e95d2cbb7bc3773d2289479947"} Oct 30 09:28:14.490329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490202 4920 scope.go:122] "RemoveContainer" containerID="882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56" Oct 30 09:28:14.490675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.490650 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88" pod "1e5bbb9b-3dc4-4795-9b56-17ae03add46c" (UID: "1e5bbb9b-3dc4-4795-9b56-17ae03add46c"). InnerVolumeSpecName "kube-api-access-sfc88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:14.491972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.491951 4920 generic.go:356] "Generic (PLEG): container finished" podID="7bca8563-e142-4c4d-852d-f68e56057a64" containerID="f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216" exitCode=0 Oct 30 09:28:14.492081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.491989 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" Oct 30 09:28:14.492081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.492026 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" event={"ID":"7bca8563-e142-4c4d-852d-f68e56057a64","Type":"ContainerDied","Data":"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216"} Oct 30 09:28:14.492081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.492069 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr" event={"ID":"7bca8563-e142-4c4d-852d-f68e56057a64","Type":"ContainerDied","Data":"211e193be767ab63916ae7ca79f4d1ab12c328c16160cfcddcb7f918be2e4a32"} Oct 30 09:28:14.534746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.534397 4920 scope.go:122] "RemoveContainer" containerID="882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56" Oct 30 09:28:14.535196 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:14.535158 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56\": container with ID starting with 882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56 not found: ID does not exist" containerID="882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56" Oct 30 09:28:14.535371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.535199 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56"} err="failed to get container status \"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56\": rpc error: code = NotFound desc = could not find container \"882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56\": container with ID starting with 882e806a3cd934cab35a6e6c6db2cd516b58247d76b05af90246bc23c4773f56 not found: ID does not exist" Oct 30 09:28:14.535371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.535242 4920 scope.go:122] "RemoveContainer" containerID="f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216" Oct 30 09:28:14.564324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.564300 4920 scope.go:122] "RemoveContainer" containerID="f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216" Oct 30 09:28:14.564695 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:14.564671 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216\": container with ID starting with f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216 not found: ID does not exist" containerID="f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216" Oct 30 09:28:14.564769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.564709 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216"} err="failed to get container status \"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216\": rpc error: code = NotFound desc = could not find container \"f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216\": container with ID starting with f93b1b53f2af75c158e254f0cfc3f8f4b6e7d9b3a57692b898f27001b9140216 not found: ID does not exist" Oct 30 09:28:14.589715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.589689 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\") pod \"7bca8563-e142-4c4d-852d-f68e56057a64\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " Oct 30 09:28:14.589852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.589736 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\" (UniqueName: \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\") pod \"7bca8563-e142-4c4d-852d-f68e56057a64\" (UID: \"7bca8563-e142-4c4d-852d-f68e56057a64\") " Oct 30 09:28:14.590147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.590109 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume" pod "7bca8563-e142-4c4d-852d-f68e56057a64" (UID: "7bca8563-e142-4c4d-852d-f68e56057a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:14.590411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.590388 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bca8563-e142-4c4d-852d-f68e56057a64-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:14.590411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.590410 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfc88\" (UniqueName: \"kubernetes.io/projected/1e5bbb9b-3dc4-4795-9b56-17ae03add46c-kube-api-access-sfc88\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:14.591938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.591916 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz" pod "7bca8563-e142-4c4d-852d-f68e56057a64" (UID: "7bca8563-e142-4c4d-852d-f68e56057a64"). InnerVolumeSpecName "kube-api-access-dg8gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:14.691776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.691732 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8gz\" (UniqueName: \"kubernetes.io/projected/7bca8563-e142-4c4d-852d-f68e56057a64-kube-api-access-dg8gz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:14.841606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.841570 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:28:14.853309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.853275 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-19a563-7ccb67fc55-7wnlr"] Oct 30 09:28:14.875615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.875569 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:28:14.890910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.890863 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf"] Oct 30 09:28:14.891355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:14.891323 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-19a563-6467c595c4-z2lzf" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded" Oct 30 09:28:14.891452 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:28:14.891412 4920 logging.go:55] [core] [Channel #1377 SubChannel #1379]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: operation was canceled" Oct 30 09:28:15.271768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:15.271675 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:16.400129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:16.400084 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" path="/var/lib/kubelet/pods/1e5bbb9b-3dc4-4795-9b56-17ae03add46c/volumes" Oct 30 09:28:16.400792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:16.400768 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bca8563-e142-4c4d-852d-f68e56057a64" path="/var/lib/kubelet/pods/7bca8563-e142-4c4d-852d-f68e56057a64/volumes" Oct 30 09:28:16.892794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:16.892761 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:18.546598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.546566 4920 generic.go:356] "Generic (PLEG): container finished" podID="add72d13-1e47-400e-a718-838226d97007" containerID="4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0" exitCode=0 Oct 30 09:28:18.547927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.546613 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerDied","Data":"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0"} Oct 30 09:28:18.547927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.547096 4920 scope.go:122] "RemoveContainer" containerID="4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0" Oct 30 09:28:18.549065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.549028 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:18.549136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.549122 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:18.549182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.549168 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:18.706325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:18.706282 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:19.259736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.256451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:19.374526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.370937 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:19.556204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.556174 4920 generic.go:356] "Generic (PLEG): container finished" podID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerID="ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4" exitCode=0 Oct 30 09:28:19.556751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.556247 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerDied","Data":"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4"} Oct 30 09:28:19.556836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.556816 4920 scope.go:122] "RemoveContainer" containerID="ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4" Oct 30 09:28:19.558559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.558534 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerStarted","Data":"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229"} Oct 30 09:28:19.558675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.558560 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:19.558675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.558641 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:19.558778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:19.558679 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:20.387235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:20.387009 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:28:20.388744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:20.388709 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 09:28:20.388888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:20.388774 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 09:28:20.391354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:20.391312 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:20.568713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:20.568675 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerStarted","Data":"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047"} Oct 30 09:28:21.171195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.171158 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:28:21.171420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.171360 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 09:28:21.171420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.171402 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 09:28:21.387985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.387947 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 09:28:21.388164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.388002 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 09:28:21.952240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:21.951999 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:22.182236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.181527 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 09:28:22.182236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.181585 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 09:28:22.189176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.188925 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:22.290120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.290085 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:22.387921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.387887 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 09:28:22.388096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.387936 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 09:28:22.389403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:22.389368 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:23.171516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:23.171478 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 09:28:23.172032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:23.171536 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 09:28:23.387874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:23.387839 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 09:28:23.388067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:23.387894 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 09:28:23.665626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:23.665581 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:24.171173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:24.171135 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 09:28:24.171360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:24.171192 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 09:28:24.387798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:24.387761 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 09:28:24.388202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:24.387818 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 09:28:24.446317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:24.445306 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:28:25.158305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.158271 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:28:25.159137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159106 4920 cpu_manager.go:414] "containerMap: removing container" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:25.159137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159129 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159161 4920 cpu_manager.go:414] "containerMap: removing container" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159166 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159189 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7bca8563-e142-4c4d-852d-f68e56057a64" containerName="payload-pre-processing" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159195 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bca8563-e142-4c4d-852d-f68e56057a64" containerName="payload-pre-processing" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159206 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerName="payload-processing" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159227 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerName="payload-processing" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159353 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5bbb9b-3dc4-4795-9b56-17ae03add46c" containerName="payload-processing" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159365 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:25.159395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.159378 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bca8563-e142-4c4d-852d-f68e56057a64" containerName="payload-pre-processing" Oct 30 09:28:25.163075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.163054 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.171104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.171081 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 09:28:25.171199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.171129 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 09:28:25.179139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.179110 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.180297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.180253 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqpg\" (UniqueName: \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.181696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.181672 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.182785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.182676 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.188892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.186341 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:28:25.213276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.203378 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d87c85-serving-cert\"" Oct 30 09:28:25.252336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.252303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:25.283828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.283799 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.284108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.284093 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqpg\" (UniqueName: \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.284261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.284246 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.284375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.284365 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.285047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.285029 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.285239 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:25.285208 4920 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-serving-cert: secret "maas-api-e2e-ipp-a-d87c85-serving-cert" not found Oct 30 09:28:25.285364 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:25.285356 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls podName:423a28f5-0f1b-4718-9126-d94482895e2d nodeName:}" failed. No retries permitted until 2026-10-30 09:28:25.785341026 +0000 UTC m=+2069.508768148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls") pod "maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" (UID: "423a28f5-0f1b-4718-9126-d94482895e2d") : secret "maas-api-e2e-ipp-a-d87c85-serving-cert" not found Oct 30 09:28:25.289721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.289698 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.331920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.331885 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqpg\" (UniqueName: \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.370237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.370181 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:25.394856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.394819 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:28:25.397089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.397062 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:28:25.491191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.491109 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:25.792578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.792540 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:25.795172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:25.795138 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:26.079325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.079200 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:26.330331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.325603 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:28:26.330331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.327694 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:28:26.330564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.330409 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:28:26.401740 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:28:26.401707 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423a28f5_0f1b_4718_9126_d94482895e2d.slice/crio-871e8dd78db6a79cf48acad3ec01ac03a7923861688e4a3e9a4ec232f0a0cf87 WatchSource:0}: Error finding container 871e8dd78db6a79cf48acad3ec01ac03a7923861688e4a3e9a4ec232f0a0cf87: Status 404 returned error can't find the container with id 871e8dd78db6a79cf48acad3ec01ac03a7923861688e4a3e9a4ec232f0a0cf87 Oct 30 09:28:26.408667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.408633 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:28:26.694530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:26.694439 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" event={"ID":"423a28f5-0f1b-4718-9126-d94482895e2d","Type":"ContainerStarted","Data":"871e8dd78db6a79cf48acad3ec01ac03a7923861688e4a3e9a4ec232f0a0cf87"} Oct 30 09:28:27.046249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:27.030987 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:27.704237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:27.704186 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" event={"ID":"423a28f5-0f1b-4718-9126-d94482895e2d","Type":"ContainerStarted","Data":"dc8c2747d8065c055cd90701a3284cac9a9107f34abc5fdfcbf9c0a34fd84bc1"} Oct 30 09:28:27.704643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:27.704272 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:27.744261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:27.744190 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" podStartSLOduration=2.4515940880000002 podStartE2EDuration="2.744139474s" totalImagesPullingTime="292.545386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:27.743632786 +0000 UTC m=+2071.467059930" watchObservedRunningTime="2026-10-30 09:28:27.744139474 +0000 UTC m=+2071.467566616" Oct 30 09:28:28.468800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:28.468758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:30.053071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:30.053040 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:31.140089 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:31.140044 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:28:31.873944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:31.873380 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:28:31.873944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:31.873670 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" podUID="a335af27-f817-4aca-ac59-37785b33927e" containerName="maas-api" containerID="cri-o://d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946" gracePeriod=30 Oct 30 09:28:31.924977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:31.924938 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:32.389244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.388561 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:28:32.470678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.470570 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:32.478110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.478082 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\") pod \"a335af27-f817-4aca-ac59-37785b33927e\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " Oct 30 09:28:32.478306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.478286 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\") pod \"a335af27-f817-4aca-ac59-37785b33927e\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " Oct 30 09:28:32.478393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.478360 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\") pod \"a335af27-f817-4aca-ac59-37785b33927e\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " Oct 30 09:28:32.478621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.478580 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca" pod "a335af27-f817-4aca-ac59-37785b33927e" (UID: "a335af27-f817-4aca-ac59-37785b33927e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:32.478768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.478747 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\" (UniqueName: \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\") pod \"a335af27-f817-4aca-ac59-37785b33927e\" (UID: \"a335af27-f817-4aca-ac59-37785b33927e\") " Oct 30 09:28:32.479383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.479364 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a335af27-f817-4aca-ac59-37785b33927e-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:32.483462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.483431 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w" pod "a335af27-f817-4aca-ac59-37785b33927e" (UID: "a335af27-f817-4aca-ac59-37785b33927e"). InnerVolumeSpecName "kube-api-access-dxf5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:32.484663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.484641 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs" pod "a335af27-f817-4aca-ac59-37785b33927e" (UID: "a335af27-f817-4aca-ac59-37785b33927e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:32.486388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.486366 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls" pod "a335af27-f817-4aca-ac59-37785b33927e" (UID: "a335af27-f817-4aca-ac59-37785b33927e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:32.570676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.570643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:32.580654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.580619 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:32.580654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.580655 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a335af27-f817-4aca-ac59-37785b33927e-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:32.580819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.580688 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf5w\" (UniqueName: \"kubernetes.io/projected/a335af27-f817-4aca-ac59-37785b33927e-kube-api-access-dxf5w\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:32.672667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.672628 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:32.760197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.760097 4920 generic.go:356] "Generic (PLEG): container finished" podID="a335af27-f817-4aca-ac59-37785b33927e" containerID="d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946" exitCode=0 Oct 30 09:28:32.760197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.760151 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" Oct 30 09:28:32.760197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.760177 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" event={"ID":"a335af27-f817-4aca-ac59-37785b33927e","Type":"ContainerDied","Data":"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946"} Oct 30 09:28:32.760454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.760249 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" event={"ID":"a335af27-f817-4aca-ac59-37785b33927e","Type":"ContainerDied","Data":"867ab7438db18a18c5a429d249550bab896ba4f921680410f4ecb8e3857329c6"} Oct 30 09:28:32.760454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.760286 4920 scope.go:122] "RemoveContainer" containerID="d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946" Oct 30 09:28:32.773111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.773091 4920 scope.go:122] "RemoveContainer" containerID="d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946" Oct 30 09:28:32.773461 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:32.773438 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946\": container with ID starting with d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946 not found: ID does not exist" containerID="d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946" Oct 30 09:28:32.773542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.773468 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946"} err="failed to get container status \"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946\": rpc error: code = NotFound desc = could not find container \"d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946\": container with ID starting with d894e34b09a8d0b99eb9803dd3bb69f233d9464534280f796997113b63e13946 not found: ID does not exist" Oct 30 09:28:32.814871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.814838 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:28:32.820673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:32.820638 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4"] Oct 30 09:28:33.108984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:33.108950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:33.719755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:33.719728 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:28:34.399290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:34.399250 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a335af27-f817-4aca-ac59-37785b33927e" path="/var/lib/kubelet/pods/a335af27-f817-4aca-ac59-37785b33927e/volumes" Oct 30 09:28:34.478029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:34.477998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:35.231615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:35.231568 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-19a563-54c98c6b6-t7rm4" podUID="a335af27-f817-4aca-ac59-37785b33927e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.182:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:28:36.269957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:36.269924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:38.011254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:38.011199 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:39.338563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.338532 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:39.447783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.447747 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:39.621830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.621738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:39.671234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.671179 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:28:39.671745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.671710 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" containerID="cri-o://f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa" gracePeriod=30 Oct 30 09:28:39.736278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:39.735249 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:40.954088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:40.954041 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:41.073397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:41.073362 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:42.233354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:42.233299 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:43.606511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:43.606469 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:28:43.614966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:43.614927 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" podUID="36686c13-752b-4466-a4b8-3582c26a2ae9" containerName="maas-api" containerID="cri-o://2db4cf3b521a18c6e954cb3a9d8b796ee3e64dc5540a31a301c9005a6af44eeb" gracePeriod=30 Oct 30 09:28:43.668933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:43.668889 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:44.094349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.093075 4920 generic.go:356] "Generic (PLEG): container finished" podID="36686c13-752b-4466-a4b8-3582c26a2ae9" containerID="2db4cf3b521a18c6e954cb3a9d8b796ee3e64dc5540a31a301c9005a6af44eeb" exitCode=0 Oct 30 09:28:44.094349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.093209 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" event={"ID":"36686c13-752b-4466-a4b8-3582c26a2ae9","Type":"ContainerDied","Data":"2db4cf3b521a18c6e954cb3a9d8b796ee3e64dc5540a31a301c9005a6af44eeb"} Oct 30 09:28:44.162676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.162640 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:28:44.276139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.276097 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\") pod \"36686c13-752b-4466-a4b8-3582c26a2ae9\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " Oct 30 09:28:44.276469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.276448 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\") pod \"36686c13-752b-4466-a4b8-3582c26a2ae9\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " Oct 30 09:28:44.276832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.276814 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\" (UniqueName: \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\") pod \"36686c13-752b-4466-a4b8-3582c26a2ae9\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " Oct 30 09:28:44.276975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.276961 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\") pod \"36686c13-752b-4466-a4b8-3582c26a2ae9\" (UID: \"36686c13-752b-4466-a4b8-3582c26a2ae9\") " Oct 30 09:28:44.278798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.278038 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca" pod "36686c13-752b-4466-a4b8-3582c26a2ae9" (UID: "36686c13-752b-4466-a4b8-3582c26a2ae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:44.281243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.281116 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5" pod "36686c13-752b-4466-a4b8-3582c26a2ae9" (UID: "36686c13-752b-4466-a4b8-3582c26a2ae9"). InnerVolumeSpecName "kube-api-access-zc8j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:44.281674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.281651 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs" pod "36686c13-752b-4466-a4b8-3582c26a2ae9" (UID: "36686c13-752b-4466-a4b8-3582c26a2ae9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:44.284365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.284163 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls" pod "36686c13-752b-4466-a4b8-3582c26a2ae9" (UID: "36686c13-752b-4466-a4b8-3582c26a2ae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:44.291924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.291901 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:44.379996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.379909 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.379996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.379944 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36686c13-752b-4466-a4b8-3582c26a2ae9-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.379996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.379959 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8j5\" (UniqueName: \"kubernetes.io/projected/36686c13-752b-4466-a4b8-3582c26a2ae9-kube-api-access-zc8j5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.379996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.379973 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36686c13-752b-4466-a4b8-3582c26a2ae9-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:44.392416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.392379 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:44.492351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.492310 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:44.948509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.948322 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:44.991626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:44.991596 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:28:45.089901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.089850 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.089929 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.089993 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090049 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090112 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090148 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090182 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090354 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090441 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\") pod \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\" (UID: \"76de7ead-1551-4ba2-886d-8f44e1bd69e9\") " Oct 30 09:28:45.090806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.090764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:45.091338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.091311 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.091542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.091516 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:45.091599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.091550 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:45.091839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.091816 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:45.091906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.091878 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:45.116190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.116144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:45.117067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.117034 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:28:45.117192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.117166 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "kube-api-access-n2cpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:45.117406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.117339 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy" pod "76de7ead-1551-4ba2-886d-8f44e1bd69e9" (UID: "76de7ead-1551-4ba2-886d-8f44e1bd69e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:45.123402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.123368 4920 generic.go:356] "Generic (PLEG): container finished" podID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerID="f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa" exitCode=0 Oct 30 09:28:45.123544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.123454 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" Oct 30 09:28:45.123544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.123500 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerDied","Data":"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa"} Oct 30 09:28:45.123655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.123591 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn" event={"ID":"76de7ead-1551-4ba2-886d-8f44e1bd69e9","Type":"ContainerDied","Data":"5446b815202f7864288b21ea1da8205ce2b2f1c384850364e9ed771864289cba"} Oct 30 09:28:45.123655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.123620 4920 scope.go:122] "RemoveContainer" containerID="f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa" Oct 30 09:28:45.126529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.126501 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" event={"ID":"36686c13-752b-4466-a4b8-3582c26a2ae9","Type":"ContainerDied","Data":"e6e79b65f13c87f0983933aac0f3cb2b8c49bcfb1964962b963a1ea5627e7db6"} Oct 30 09:28:45.126646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.126603 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6" Oct 30 09:28:45.138698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.138678 4920 scope.go:122] "RemoveContainer" containerID="3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6" Oct 30 09:28:45.150191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.150165 4920 scope.go:122] "RemoveContainer" containerID="f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa" Oct 30 09:28:45.150550 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:45.150529 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa\": container with ID starting with f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa not found: ID does not exist" containerID="f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa" Oct 30 09:28:45.150610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.150560 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa"} err="failed to get container status \"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa\": rpc error: code = NotFound desc = could not find container \"f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa\": container with ID starting with f5fec72899d701791cd309ad70795affa5303a02f8fde8c42897d1bfdfaeb2fa not found: ID does not exist" Oct 30 09:28:45.150610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.150579 4920 scope.go:122] "RemoveContainer" containerID="3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6" Oct 30 09:28:45.150933 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:45.150911 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6\": container with ID starting with 3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6 not found: ID does not exist" containerID="3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6" Oct 30 09:28:45.151023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.150938 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6"} err="failed to get container status \"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6\": rpc error: code = NotFound desc = could not find container \"3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6\": container with ID starting with 3979d481a531fb2748daf56b127498c67df852123e5faeaac077e65290f8bbd6 not found: ID does not exist" Oct 30 09:28:45.151023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.150954 4920 scope.go:122] "RemoveContainer" containerID="2db4cf3b521a18c6e954cb3a9d8b796ee3e64dc5540a31a301c9005a6af44eeb" Oct 30 09:28:45.175944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.175912 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:28:45.180301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.180273 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-20f71c-5c4cc9f8cb-kqcp6"] Oct 30 09:28:45.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192586 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192615 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192624 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192637 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192646 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192654 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192663 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2cpj\" (UniqueName: \"kubernetes.io/projected/76de7ead-1551-4ba2-886d-8f44e1bd69e9-kube-api-access-n2cpj\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.192835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.192673 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de7ead-1551-4ba2-886d-8f44e1bd69e9-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:45.216991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.216909 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:28:45.222705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.222665 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-19a563-openshift-default-6d97cdb7d9-hkpjn"] Oct 30 09:28:45.247422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.247377 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:28:45.247589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:45.247461 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:28:46.360150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:46.360108 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:46.404174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:46.404138 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36686c13-752b-4466-a4b8-3582c26a2ae9" path="/var/lib/kubelet/pods/36686c13-752b-4466-a4b8-3582c26a2ae9/volumes" Oct 30 09:28:46.404675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:46.404651 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" path="/var/lib/kubelet/pods/76de7ead-1551-4ba2-886d-8f44e1bd69e9/volumes" Oct 30 09:28:47.148465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.148388 4920 generic.go:356] "Generic (PLEG): container finished" podID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerID="2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700" exitCode=0 Oct 30 09:28:47.148660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.148464 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerDied","Data":"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700"} Oct 30 09:28:47.148951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.148935 4920 scope.go:122] "RemoveContainer" containerID="2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700" Oct 30 09:28:47.150557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.150525 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:47.150671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.150609 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:47.150671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.150660 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:47.779242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:47.776115 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:48.159578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:48.159536 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerStarted","Data":"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7"} Oct 30 09:28:48.167268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:48.167243 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:28:48.167457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:48.167441 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:48.167514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:48.167480 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:49.176365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:49.176268 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:49.176915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:49.176406 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:49.363884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:49.363844 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:50.168291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:50.168252 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:50.168502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:50.168334 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:50.331189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:50.331145 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:50.922541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:50.922476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:51.167975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.167939 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:51.168158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.168007 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:51.338390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.338355 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:51.445536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.445502 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:51.669474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.669391 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:28:51.669668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.669637 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" containerID="cri-o://df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c" gracePeriod=30 Oct 30 09:28:51.709624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.709588 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:51.860970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.860924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:51.962142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:51.962061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:52.062233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.062194 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:52.170258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.169342 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:52.170258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.169405 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:52.513247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.499894 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5"] Oct 30 09:28:52.514817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.514785 4920 cpu_manager.go:414] "containerMap: removing container" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.514984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.514970 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.515101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515089 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a335af27-f817-4aca-ac59-37785b33927e" containerName="maas-api" Oct 30 09:28:52.515185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515174 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a335af27-f817-4aca-ac59-37785b33927e" containerName="maas-api" Oct 30 09:28:52.515322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515309 4920 cpu_manager.go:414] "containerMap: removing container" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.515412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515401 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.515509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515499 4920 cpu_manager.go:414] "containerMap: removing container" podUID="36686c13-752b-4466-a4b8-3582c26a2ae9" containerName="maas-api" Oct 30 09:28:52.515589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515579 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36686c13-752b-4466-a4b8-3582c26a2ae9" containerName="maas-api" Oct 30 09:28:52.515841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515816 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a335af27-f817-4aca-ac59-37785b33927e" containerName="maas-api" Oct 30 09:28:52.515960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.515947 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfebb963-2069-47e8-a15c-c6515b0af8a2" containerName="istio-proxy" Oct 30 09:28:52.516049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.516039 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.516130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.516121 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="36686c13-752b-4466-a4b8-3582c26a2ae9" containerName="maas-api" Oct 30 09:28:52.516252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.516239 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="76de7ead-1551-4ba2-886d-8f44e1bd69e9" containerName="istio-proxy" Oct 30 09:28:52.531629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.531596 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.548526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.548488 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:28:52.562242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.549478 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:28:52.562484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.560166 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5"] Oct 30 09:28:52.690939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.690896 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.691166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.691142 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.691255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.691225 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fs9\" (UniqueName: \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.792352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.792317 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.792521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.792369 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fs9\" (UniqueName: \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.792521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.792486 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.793135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.793112 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.794777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.794755 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.814718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.814685 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fs9\" (UniqueName: \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\") pod \"maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.886928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.886887 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:52.902802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:52.902669 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:53.077619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.077523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:53.168309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.168270 4920 patch_prober.go:29] interesting pod/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 09:28:53.168551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.168332 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 09:28:53.243401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.243364 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5"] Oct 30 09:28:53.316081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.313735 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:53.331016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.330926 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:28:53.340574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.340541 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.355923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.355874 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-15e1cc-openshift-default-dockercfg-wpsgw\"" Oct 30 09:28:53.363763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.363735 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:28:53.432875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.432835 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:53.538884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.538843 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539020 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539079 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539176 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539346 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539379 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x67s\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539502 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539563 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.539800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.539696 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.556939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.556900 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643442 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643506 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643586 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643648 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643715 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643751 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643781 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643865 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.643890 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x67s\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.644666 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.645838 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.646488 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.646718 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.647250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.646947 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.674749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.674696 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.675944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.675922 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.679840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.679782 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x67s\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.691680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.691645 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\") pod \"e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:53.725720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:53.724477 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:54.099748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.099534 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:28:54.103559 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:28:54.103527 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acf74bf_7ee2_474d_af8e_b0d857263c6b.slice/crio-96ae90ff5b86f8e760cae3609747c88fcc383e74eaf6e8db073daade6584f77a WatchSource:0}: Error finding container 96ae90ff5b86f8e760cae3609747c88fcc383e74eaf6e8db073daade6584f77a: Status 404 returned error can't find the container with id 96ae90ff5b86f8e760cae3609747c88fcc383e74eaf6e8db073daade6584f77a Oct 30 09:28:54.106963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.106922 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:54.107053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.107013 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:54.107094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.107056 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:28:54.203116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.201843 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:28:54.203286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.203261 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:28:54.204632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.204607 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:28:54.258237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.257789 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" event={"ID":"1f17c05f-a9b4-427a-b4bc-ffcd01128640","Type":"ContainerStarted","Data":"112a7298aa7fcb0bad317ca5a44005d60b62ad2f025b067dc39eba98a68fccf3"} Oct 30 09:28:54.258237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.257842 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" event={"ID":"1f17c05f-a9b4-427a-b4bc-ffcd01128640","Type":"ContainerStarted","Data":"b5775c180928048338f90ce0960bc14ff3078f369c90a5001d5fdcc60c060da2"} Oct 30 09:28:54.260874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.260495 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerStarted","Data":"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b"} Oct 30 09:28:54.260874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.260534 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerStarted","Data":"96ae90ff5b86f8e760cae3609747c88fcc383e74eaf6e8db073daade6584f77a"} Oct 30 09:28:54.377811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.377707 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podStartSLOduration=1.377693413 podStartE2EDuration="1.377693413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:54.376087819 +0000 UTC m=+2098.099514978" watchObservedRunningTime="2026-10-30 09:28:54.377693413 +0000 UTC m=+2098.101120555" Oct 30 09:28:54.380066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.380024 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" podStartSLOduration=2.380013913 podStartE2EDuration="2.380013913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:54.33362839 +0000 UTC m=+2098.057055534" watchObservedRunningTime="2026-10-30 09:28:54.380013913 +0000 UTC m=+2098.103441055" Oct 30 09:28:54.727452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.727365 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:54.983630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:54.983546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:55.270468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:55.270364 4920 generic.go:356] "Generic (PLEG): container finished" podID="1f17c05f-a9b4-427a-b4bc-ffcd01128640" containerID="112a7298aa7fcb0bad317ca5a44005d60b62ad2f025b067dc39eba98a68fccf3" exitCode=0 Oct 30 09:28:55.270637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:55.270458 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" event={"ID":"1f17c05f-a9b4-427a-b4bc-ffcd01128640","Type":"ContainerDied","Data":"112a7298aa7fcb0bad317ca5a44005d60b62ad2f025b067dc39eba98a68fccf3"} Oct 30 09:28:55.733175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:55.733144 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:56.278466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.278435 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:56.279627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.279602 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:28:56.516393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.516361 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:56.617429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.617390 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\") pod \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " Oct 30 09:28:56.617613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.617581 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\") pod \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " Oct 30 09:28:56.617677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.617664 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\" (UniqueName: \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\") pod \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\" (UID: \"1f17c05f-a9b4-427a-b4bc-ffcd01128640\") " Oct 30 09:28:56.617849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.617821 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca" pod "1f17c05f-a9b4-427a-b4bc-ffcd01128640" (UID: "1f17c05f-a9b4-427a-b4bc-ffcd01128640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:56.618272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.618247 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:56.619726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.619702 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca" pod "1f17c05f-a9b4-427a-b4bc-ffcd01128640" (UID: "1f17c05f-a9b4-427a-b4bc-ffcd01128640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:28:56.619811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.619778 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9" pod "1f17c05f-a9b4-427a-b4bc-ffcd01128640" (UID: "1f17c05f-a9b4-427a-b4bc-ffcd01128640"). InnerVolumeSpecName "kube-api-access-h6fs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:56.720057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.720018 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f17c05f-a9b4-427a-b4bc-ffcd01128640-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:56.720057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.720050 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fs9\" (UniqueName: \"kubernetes.io/projected/1f17c05f-a9b4-427a-b4bc-ffcd01128640-kube-api-access-h6fs9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:56.768652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.768589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:56.864060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.864028 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:56.937408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:56.937382 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:28:57.023903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.023870 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.023965 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024010 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024082 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024140 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024238 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024280 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024342 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.024615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.024435 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\") pod \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\" (UID: \"c36a6111-7df9-49c8-8080-ef13d03d4ab3\") " Oct 30 09:28:57.025455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.025425 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:57.025850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.025826 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:57.026031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026005 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:57.026252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026201 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:57.026555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026491 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:57.026555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026520 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:28:57.026680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026594 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "kube-api-access-c48kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:57.026801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.026776 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:57.027440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.027417 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy" pod "c36a6111-7df9-49c8-8080-ef13d03d4ab3" (UID: "c36a6111-7df9-49c8-8080-ef13d03d4ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:28:57.126400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126368 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126394 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126404 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126412 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126420 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126429 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36a6111-7df9-49c8-8080-ef13d03d4ab3-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126437 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126445 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36a6111-7df9-49c8-8080-ef13d03d4ab3-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.126770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.126453 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48kn\" (UniqueName: \"kubernetes.io/projected/c36a6111-7df9-49c8-8080-ef13d03d4ab3-kube-api-access-c48kn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.211613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.211530 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:57.310449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.310424 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" Oct 30 09:28:57.310706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.310472 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5" event={"ID":"1f17c05f-a9b4-427a-b4bc-ffcd01128640","Type":"ContainerDied","Data":"b5775c180928048338f90ce0960bc14ff3078f369c90a5001d5fdcc60c060da2"} Oct 30 09:28:57.310878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.310744 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5775c180928048338f90ce0960bc14ff3078f369c90a5001d5fdcc60c060da2" Oct 30 09:28:57.313058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.312873 4920 generic.go:356] "Generic (PLEG): container finished" podID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerID="df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c" exitCode=0 Oct 30 09:28:57.313058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.312949 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" Oct 30 09:28:57.313467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.313372 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerDied","Data":"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c"} Oct 30 09:28:57.313467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.313436 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh" event={"ID":"c36a6111-7df9-49c8-8080-ef13d03d4ab3","Type":"ContainerDied","Data":"f160baa7736c2108b779d4b8c00e38da8586374f55bfd1b2c644688e6d999a6b"} Oct 30 09:28:57.314087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.313877 4920 scope.go:122] "RemoveContainer" containerID="df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c" Oct 30 09:28:57.342522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.342493 4920 scope.go:122] "RemoveContainer" containerID="00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3" Oct 30 09:28:57.352867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.352846 4920 scope.go:122] "RemoveContainer" containerID="df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c" Oct 30 09:28:57.353206 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:57.353181 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c\": container with ID starting with df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c not found: ID does not exist" containerID="df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c" Oct 30 09:28:57.353296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.353258 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c"} err="failed to get container status \"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c\": rpc error: code = NotFound desc = could not find container \"df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c\": container with ID starting with df7cb1658c871ec2bf376a64a3d1642a79047077682b488f848b02b5ddbdd22c not found: ID does not exist" Oct 30 09:28:57.353296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.353281 4920 scope.go:122] "RemoveContainer" containerID="00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3" Oct 30 09:28:57.353582 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:28:57.353564 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3\": container with ID starting with 00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3 not found: ID does not exist" containerID="00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3" Oct 30 09:28:57.353630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.353589 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3"} err="failed to get container status \"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3\": rpc error: code = NotFound desc = could not find container \"00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3\": container with ID starting with 00b0ea41ce9b30384b7fa66e31f36e2fb32c304ec28e6c297a9cf16823aa41b3 not found: ID does not exist" Oct 30 09:28:57.386006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.385968 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:28:57.390478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:57.390449 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-20f71c-openshift-default-fc6f649b5-qvxgh"] Oct 30 09:28:58.402935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:58.402898 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" path="/var/lib/kubelet/pods/c36a6111-7df9-49c8-8080-ef13d03d4ab3/volumes" Oct 30 09:28:58.433778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:58.433726 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:58.531824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:58.531792 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:59.168593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:59.168537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:59.705530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:59.705478 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:28:59.805243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:28:59.805198 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:00.803174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:00.803116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:00.872893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:00.872856 4920 scope.go:122] "RemoveContainer" containerID="29324f22eda4f9edeeaec4660291a465ec47cba407388de58588173d878fcb77" Oct 30 09:29:01.135975 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:01.135876 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:29:02.303307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:02.303271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:03.054396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.054365 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9"] Oct 30 09:29:03.055128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055107 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1f17c05f-a9b4-427a-b4bc-ffcd01128640" containerName="revoke-keys" Oct 30 09:29:03.055128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055129 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f17c05f-a9b4-427a-b4bc-ffcd01128640" containerName="revoke-keys" Oct 30 09:29:03.055340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055153 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.055340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055162 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.055340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055203 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.055340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055211 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.055560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055386 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f17c05f-a9b4-427a-b4bc-ffcd01128640" containerName="revoke-keys" Oct 30 09:29:03.055560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055403 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.055560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.055420 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36a6111-7df9-49c8-8080-ef13d03d4ab3" containerName="istio-proxy" Oct 30 09:29:03.063098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.063071 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.081075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.077328 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9"] Oct 30 09:29:03.083381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.083359 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:03.083623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.083609 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:03.110146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.106568 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:03.185100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.185066 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:29:03.233281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.232020 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv6v\" (UniqueName: \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.233281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.232103 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.233281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.232159 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.254286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.238284 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:03.277956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.277829 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:29:03.333425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.333316 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv6v\" (UniqueName: \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.333425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.333414 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.333938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.333482 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.334281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.334256 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.335926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.335904 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.352909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.352877 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv6v\" (UniqueName: \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\") pod \"maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.400033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.400000 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:03.779720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:03.779684 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9"] Oct 30 09:29:03.782420 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:03.782390 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5344ce3b_f736_4196_9e3b_e7879b40d7a9.slice/crio-5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d WatchSource:0}: Error finding container 5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d: Status 404 returned error can't find the container with id 5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d Oct 30 09:29:04.257946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:04.257912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:04.401448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:04.401355 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" event={"ID":"5344ce3b-f736-4196-9e3b-e7879b40d7a9","Type":"ContainerStarted","Data":"4c10f58cd89f47146b23c79e2b02de4768b55632e309d8ee5e55a2957dd0b0c8"} Oct 30 09:29:04.401448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:04.401421 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" event={"ID":"5344ce3b-f736-4196-9e3b-e7879b40d7a9","Type":"ContainerStarted","Data":"5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d"} Oct 30 09:29:04.439477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:04.439422 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" podStartSLOduration=1.439406318 podStartE2EDuration="1.439406318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:04.437770733 +0000 UTC m=+2108.161197908" watchObservedRunningTime="2026-10-30 09:29:04.439406318 +0000 UTC m=+2108.162833460" Oct 30 09:29:05.030806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:05.029982 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:29:05.413598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:05.413562 4920 generic.go:356] "Generic (PLEG): container finished" podID="5344ce3b-f736-4196-9e3b-e7879b40d7a9" containerID="4c10f58cd89f47146b23c79e2b02de4768b55632e309d8ee5e55a2957dd0b0c8" exitCode=0 Oct 30 09:29:05.414036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:05.413622 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" event={"ID":"5344ce3b-f736-4196-9e3b-e7879b40d7a9","Type":"ContainerDied","Data":"4c10f58cd89f47146b23c79e2b02de4768b55632e309d8ee5e55a2957dd0b0c8"} Oct 30 09:29:06.237884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.237840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:06.827088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.827057 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:06.868493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.868409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:06.918599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.918569 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\") pod \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " Oct 30 09:29:06.918795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.918610 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\" (UniqueName: \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\") pod \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " Oct 30 09:29:06.918795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.918677 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\") pod \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\" (UID: \"5344ce3b-f736-4196-9e3b-e7879b40d7a9\") " Oct 30 09:29:06.919176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.919143 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca" pod "5344ce3b-f736-4196-9e3b-e7879b40d7a9" (UID: "5344ce3b-f736-4196-9e3b-e7879b40d7a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:06.919393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.919376 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:06.920903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.920879 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca" pod "5344ce3b-f736-4196-9e3b-e7879b40d7a9" (UID: "5344ce3b-f736-4196-9e3b-e7879b40d7a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:06.921015 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.920915 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v" pod "5344ce3b-f736-4196-9e3b-e7879b40d7a9" (UID: "5344ce3b-f736-4196-9e3b-e7879b40d7a9"). InnerVolumeSpecName "kube-api-access-pxv6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:06.967435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:06.967391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:07.021233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.021186 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5344ce3b-f736-4196-9e3b-e7879b40d7a9-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:07.021419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.021261 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxv6v\" (UniqueName: \"kubernetes.io/projected/5344ce3b-f736-4196-9e3b-e7879b40d7a9-kube-api-access-pxv6v\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:07.073480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.073442 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:07.471463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.471400 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:07.482874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.482831 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" event={"ID":"5344ce3b-f736-4196-9e3b-e7879b40d7a9","Type":"ContainerDied","Data":"5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d"} Oct 30 09:29:07.483088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.482944 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b78f61dac862b1a9147f1482714aa4505df2c4b00067c2d490c38d375cd831d" Oct 30 09:29:07.483088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:07.483030 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9" Oct 30 09:29:08.750102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:08.750004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:10.241925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:10.241889 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:11.819390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:11.819329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:13.006927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.006895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:13.148184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.148149 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:13.416916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.416868 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:29:13.418263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.418234 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5344ce3b-f736-4196-9e3b-e7879b40d7a9" containerName="revoke-keys" Oct 30 09:29:13.418662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.418266 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5344ce3b-f736-4196-9e3b-e7879b40d7a9" containerName="revoke-keys" Oct 30 09:29:13.419037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.419017 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5344ce3b-f736-4196-9e3b-e7879b40d7a9" containerName="revoke-keys" Oct 30 09:29:13.424437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.424416 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.434713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.434688 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:29:13.441827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.441800 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d87c85-dockercfg-x4plm\"" Oct 30 09:29:13.441933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.441900 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d87c85\"" Oct 30 09:29:13.575245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.575131 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.579445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.579409 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzt4g\" (UniqueName: \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.639660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.639622 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:29:13.648564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.648532 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:29:13.648807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.648662 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.709493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.709393 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.710430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.710322 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.710615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.710592 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.710714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.710697 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7gh\" (UniqueName: \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.710795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.710780 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzt4g\" (UniqueName: \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.730496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.730466 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzt4g\" (UniqueName: \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.740828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.740796 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:13.806513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.806478 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:13.813331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.813297 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.813494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.813400 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7gh\" (UniqueName: \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.813983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.813960 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.841968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.841890 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7gh\" (UniqueName: \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\") pod \"payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:13.905236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.905116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:13.981719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:13.981635 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:14.073271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.072479 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:29:14.073710 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:14.073472 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8d9b27_63b9_4ea4_a889_b732ce46498e.slice/crio-7c9fb2e8e6c19845cc249aec39a129d39bec7e5c02e6bb5f3dad8fc9f7a3268c WatchSource:0}: Error finding container 7c9fb2e8e6c19845cc249aec39a129d39bec7e5c02e6bb5f3dad8fc9f7a3268c: Status 404 returned error can't find the container with id 7c9fb2e8e6c19845cc249aec39a129d39bec7e5c02e6bb5f3dad8fc9f7a3268c Oct 30 09:29:14.118200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.117156 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:29:14.120193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.117529 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" containerID="cri-o://e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771" gracePeriod=30 Oct 30 09:29:14.140023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.139988 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:14.272296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.270246 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:14.402807 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:14.402773 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c46a36_4a8d_4db2_bc02_be1e87f52b13.slice/crio-4586b2895689f0d911eaaa760ec8e6ffcf6af9dd9c0b4c689e1de007d7630682 WatchSource:0}: Error finding container 4586b2895689f0d911eaaa760ec8e6ffcf6af9dd9c0b4c689e1de007d7630682: Status 404 returned error can't find the container with id 4586b2895689f0d911eaaa760ec8e6ffcf6af9dd9c0b4c689e1de007d7630682 Oct 30 09:29:14.403140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.403120 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:29:14.614616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.614576 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" event={"ID":"c1c46a36-4a8d-4db2-bc02-be1e87f52b13","Type":"ContainerStarted","Data":"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b"} Oct 30 09:29:14.615105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.614647 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:14.615105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.614672 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" event={"ID":"c1c46a36-4a8d-4db2-bc02-be1e87f52b13","Type":"ContainerStarted","Data":"4586b2895689f0d911eaaa760ec8e6ffcf6af9dd9c0b4c689e1de007d7630682"} Oct 30 09:29:14.616587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.616560 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" event={"ID":"6d8d9b27-63b9-4ea4-a889-b732ce46498e","Type":"ContainerStarted","Data":"986988b686842b906904407a194c953bdf34f7736ce8d183f46d49b3ae2da458"} Oct 30 09:29:14.616712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.616610 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" event={"ID":"6d8d9b27-63b9-4ea4-a889-b732ce46498e","Type":"ContainerStarted","Data":"7c9fb2e8e6c19845cc249aec39a129d39bec7e5c02e6bb5f3dad8fc9f7a3268c"} Oct 30 09:29:14.616712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.616664 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:14.654029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.653980 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" podStartSLOduration=1.6539650689999998 podStartE2EDuration="1.653965069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:14.652802103 +0000 UTC m=+2118.376229247" watchObservedRunningTime="2026-10-30 09:29:14.653965069 +0000 UTC m=+2118.377392247" Oct 30 09:29:14.695178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:14.695122 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" podStartSLOduration=1.695105124 podStartE2EDuration="1.695105124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:14.692756816 +0000 UTC m=+2118.416183960" watchObservedRunningTime="2026-10-30 09:29:14.695105124 +0000 UTC m=+2118.418532267" Oct 30 09:29:15.509338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:15.502325 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:15.896238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:15.895796 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:29:15.896238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:15.896042 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" podUID="348b3649-4752-4d7e-809d-3110c58cb8cf" containerName="maas-api" containerID="cri-o://272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9" gracePeriod=30 Oct 30 09:29:16.016244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.015592 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:16.118749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.118697 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:16.212702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.212666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:16.259023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.258981 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5"] Oct 30 09:29:16.263368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.263336 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cbd96e-4wdd5"] Oct 30 09:29:16.434299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.417622 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:29:16.479128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.474875 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\") pod \"348b3649-4752-4d7e-809d-3110c58cb8cf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " Oct 30 09:29:16.479128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.474954 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\") pod \"348b3649-4752-4d7e-809d-3110c58cb8cf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " Oct 30 09:29:16.479128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.475201 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\") pod \"348b3649-4752-4d7e-809d-3110c58cb8cf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " Oct 30 09:29:16.479128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.475257 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\" (UniqueName: \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\") pod \"348b3649-4752-4d7e-809d-3110c58cb8cf\" (UID: \"348b3649-4752-4d7e-809d-3110c58cb8cf\") " Oct 30 09:29:16.482728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.482683 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca" pod "348b3649-4752-4d7e-809d-3110c58cb8cf" (UID: "348b3649-4752-4d7e-809d-3110c58cb8cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:16.498279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.498240 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f17c05f-a9b4-427a-b4bc-ffcd01128640" path="/var/lib/kubelet/pods/1f17c05f-a9b4-427a-b4bc-ffcd01128640/volumes" Oct 30 09:29:16.501405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.499576 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls" pod "348b3649-4752-4d7e-809d-3110c58cb8cf" (UID: "348b3649-4752-4d7e-809d-3110c58cb8cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:16.502513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.502459 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb" pod "348b3649-4752-4d7e-809d-3110c58cb8cf" (UID: "348b3649-4752-4d7e-809d-3110c58cb8cf"). InnerVolumeSpecName "kube-api-access-b6fmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:16.514806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.514767 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs" pod "348b3649-4752-4d7e-809d-3110c58cb8cf" (UID: "348b3649-4752-4d7e-809d-3110c58cb8cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:16.577120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.577082 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348b3649-4752-4d7e-809d-3110c58cb8cf-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.577334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.577139 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6fmb\" (UniqueName: \"kubernetes.io/projected/348b3649-4752-4d7e-809d-3110c58cb8cf-kube-api-access-b6fmb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.577334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.577155 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.577334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.577168 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b3649-4752-4d7e-809d-3110c58cb8cf-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.639379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.639341 4920 generic.go:356] "Generic (PLEG): container finished" podID="348b3649-4752-4d7e-809d-3110c58cb8cf" containerID="272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9" exitCode=0 Oct 30 09:29:16.639561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.639400 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" Oct 30 09:29:16.639561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.639414 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" event={"ID":"348b3649-4752-4d7e-809d-3110c58cb8cf","Type":"ContainerDied","Data":"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9"} Oct 30 09:29:16.639561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.639470 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf" event={"ID":"348b3649-4752-4d7e-809d-3110c58cb8cf","Type":"ContainerDied","Data":"0d9056f18a372125d4d2fcc384fee8053ff5e5ac16e42a1aebcfb8b130fce9a9"} Oct 30 09:29:16.639561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.639496 4920 scope.go:122] "RemoveContainer" containerID="272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9" Oct 30 09:29:16.670818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.670792 4920 scope.go:122] "RemoveContainer" containerID="272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9" Oct 30 09:29:16.671171 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:16.671149 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9\": container with ID starting with 272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9 not found: ID does not exist" containerID="272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9" Oct 30 09:29:16.671296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.671195 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9"} err="failed to get container status \"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9\": rpc error: code = NotFound desc = could not find container \"272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9\": container with ID starting with 272e22564d89e49b6e74bf032b128ad4f63c279b8e92e0bc84cee88f8df44ac9 not found: ID does not exist" Oct 30 09:29:16.706834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.706713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:16.706834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.706773 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:29:16.710589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.710564 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9d39c0-5c75b54fcf-nndkf"] Oct 30 09:29:16.781696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.781659 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:29:16.782466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.782439 4920 cpu_manager.go:414] "containerMap: removing container" podUID="348b3649-4752-4d7e-809d-3110c58cb8cf" containerName="maas-api" Oct 30 09:29:16.782466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.782462 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348b3649-4752-4d7e-809d-3110c58cb8cf" containerName="maas-api" Oct 30 09:29:16.782676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.782585 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="348b3649-4752-4d7e-809d-3110c58cb8cf" containerName="maas-api" Oct 30 09:29:16.787596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.787566 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.838867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.838831 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:29:16.854798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.854761 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:16.872851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.872817 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-15e1cc-serving-cert\"" Oct 30 09:29:16.884446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.884413 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.884632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.884490 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.884632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.884543 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8j78\" (UniqueName: \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.884721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.884697 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.926131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.926094 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9"] Oct 30 09:29:16.931052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.931018 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9d39c0-bl4p9"] Oct 30 09:29:16.986293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.986165 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.986463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.986322 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.986463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.986364 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.986463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.986383 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8j78\" (UniqueName: \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.987025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.987002 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.988716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.988695 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.988793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.988748 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:16.999828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:16.999802 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:17.006038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.006013 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8j78\" (UniqueName: \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\") pod \"maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:17.103168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.103132 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:17.121375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.121273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:17.346276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.346244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:17.423827 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:17.423789 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5782e737_46bd_462d_8d79_47895448870c.slice/crio-f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0 WatchSource:0}: Error finding container f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0: Status 404 returned error can't find the container with id f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0 Oct 30 09:29:17.426686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.426662 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:29:17.649605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:17.649518 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" event={"ID":"5782e737-46bd-462d-8d79-47895448870c","Type":"ContainerStarted","Data":"f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0"} Oct 30 09:29:18.399203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.399163 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348b3649-4752-4d7e-809d-3110c58cb8cf" path="/var/lib/kubelet/pods/348b3649-4752-4d7e-809d-3110c58cb8cf/volumes" Oct 30 09:29:18.399806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.399780 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5344ce3b-f736-4196-9e3b-e7879b40d7a9" path="/var/lib/kubelet/pods/5344ce3b-f736-4196-9e3b-e7879b40d7a9/volumes" Oct 30 09:29:18.660006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.659915 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" event={"ID":"5782e737-46bd-462d-8d79-47895448870c","Type":"ContainerStarted","Data":"2c1aaec4b620ff011ccce437e53c0e149661a7a7915582cff2f296ad26fc5fca"} Oct 30 09:29:18.660449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.660029 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:18.666896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.666871 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:18.702775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:18.702719 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" podStartSLOduration=2.2698233500000002 podStartE2EDuration="2.702705449s" totalImagesPullingTime="432.882099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:18.701917333 +0000 UTC m=+2122.425344498" watchObservedRunningTime="2026-10-30 09:29:18.702705449 +0000 UTC m=+2122.426132592" Oct 30 09:29:19.414911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.414880 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:29:19.537286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537245 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537299 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537367 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537444 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537510 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537557 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537614 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537643 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537672 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\") pod \"3757924f-64e9-4d9b-a37c-45fe23eb2637\" (UID: \"3757924f-64e9-4d9b-a37c-45fe23eb2637\") " Oct 30 09:29:19.537968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.537939 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.538211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.538055 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.538406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.538371 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.538473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.538414 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.538591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.538566 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:19.539114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.539091 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757924f-64e9-4d9b-a37c-45fe23eb2637-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.539770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.539743 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.539871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.539769 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.539871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.539805 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.539871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.539824 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.541137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.541100 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.541314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.541290 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "kube-api-access-nv5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.541451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.541424 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:19.541847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.541827 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy" pod "3757924f-64e9-4d9b-a37c-45fe23eb2637" (UID: "3757924f-64e9-4d9b-a37c-45fe23eb2637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.641108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.641013 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.641108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.641045 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.641108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.641057 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5qv\" (UniqueName: \"kubernetes.io/projected/3757924f-64e9-4d9b-a37c-45fe23eb2637-kube-api-access-nv5qv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.641108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.641070 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757924f-64e9-4d9b-a37c-45fe23eb2637-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.670957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.670916 4920 generic.go:356] "Generic (PLEG): container finished" podID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerID="e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771" exitCode=0 Oct 30 09:29:19.671477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.670963 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" Oct 30 09:29:19.671477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.670966 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerDied","Data":"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771"} Oct 30 09:29:19.671477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.671038 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc" event={"ID":"3757924f-64e9-4d9b-a37c-45fe23eb2637","Type":"ContainerDied","Data":"040ecf75ddf4720ae2d9d7568c3904f8afef47d7676612cb92ea645e7c88fb8b"} Oct 30 09:29:19.671477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.671074 4920 scope.go:122] "RemoveContainer" containerID="e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771" Oct 30 09:29:19.683838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.683747 4920 scope.go:122] "RemoveContainer" containerID="671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e" Oct 30 09:29:19.694889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.694864 4920 scope.go:122] "RemoveContainer" containerID="e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771" Oct 30 09:29:19.695199 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:19.695178 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771\": container with ID starting with e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771 not found: ID does not exist" containerID="e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771" Oct 30 09:29:19.695319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.695211 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771"} err="failed to get container status \"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771\": rpc error: code = NotFound desc = could not find container \"e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771\": container with ID starting with e1f6caa3b7e19c3d8269a9b03c4a1b4ce8c6a94428e9341b7deeee3ca60cc771 not found: ID does not exist" Oct 30 09:29:19.695319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.695250 4920 scope.go:122] "RemoveContainer" containerID="671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e" Oct 30 09:29:19.695531 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:19.695512 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e\": container with ID starting with 671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e not found: ID does not exist" containerID="671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e" Oct 30 09:29:19.695586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.695538 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e"} err="failed to get container status \"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e\": rpc error: code = NotFound desc = could not find container \"671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e\": container with ID starting with 671580f322458fcb310c79b23a5f23165626b37f8f722ce45725e0a5b5b2624e not found: ID does not exist" Oct 30 09:29:19.724554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.724521 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:29:19.730500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:19.730470 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-xjmnc"] Oct 30 09:29:20.247837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:20.247798 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:20.353999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:20.353964 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:20.400478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:20.400442 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" path="/var/lib/kubelet/pods/3757924f-64e9-4d9b-a37c-45fe23eb2637/volumes" Oct 30 09:29:21.844612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:21.844579 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:23.401847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:23.401697 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:23.524239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:23.522874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:23.792434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:23.792390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:23.817228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:23.814527 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:29:23.817228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:23.814987 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" containerID="cri-o://1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047" gracePeriod=30 Oct 30 09:29:24.162613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.162533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:24.263478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.263446 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:24.496395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.496316 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:29:24.497129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.497096 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" containerID="cri-o://2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120" gracePeriod=30 Oct 30 09:29:24.520796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.520762 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:24.699474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:24.697390 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:29:25.221266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:25.221207 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:25.629353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:25.629204 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:29:25.629859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:25.629451 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:29:26.329988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:26.329951 4920 patch_prober.go:29] interesting pod/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:29:26.330172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:26.330022 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:29:27.344718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:27.344682 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:28.846949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:28.846913 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:29.119960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.119936 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:29:29.154647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154615 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154659 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154691 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154733 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154775 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154797 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.154825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154825 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154919 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.154994 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\") pod \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\" (UID: \"5f740194-fd61-4e36-b6b5-e83de6aafa9d\") " Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155053 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155091 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155331 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155428 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:29.155504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155438 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.155899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155620 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.155899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155640 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.155899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155652 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.155899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155660 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.155899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.155669 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.157078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.157053 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:29.157290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.157270 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "kube-api-access-nk2wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:29.157474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.157449 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.157474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.157459 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token" pod "5f740194-fd61-4e36-b6b5-e83de6aafa9d" (UID: "5f740194-fd61-4e36-b6b5-e83de6aafa9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:29.256822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.256785 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2wq\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-kube-api-access-nk2wq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.256822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.256816 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.256822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.256826 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.257061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.256835 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f740194-fd61-4e36-b6b5-e83de6aafa9d-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:29.768630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.768561 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:29:29.772927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.772897 4920 generic.go:356] "Generic (PLEG): container finished" podID="bf103a26-1163-4f3d-8d4a-09481a716766" containerID="2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120" exitCode=0 Oct 30 09:29:29.773087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.772936 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" Oct 30 09:29:29.773158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.772975 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerDied","Data":"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120"} Oct 30 09:29:29.773257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.773151 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln" event={"ID":"bf103a26-1163-4f3d-8d4a-09481a716766","Type":"ContainerDied","Data":"56ce5db0e9aacb935d465cedd1fc3deec0db653dfcb5df310bde929f7fa5b487"} Oct 30 09:29:29.773257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.773183 4920 scope.go:122] "RemoveContainer" containerID="2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120" Oct 30 09:29:29.775084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.775060 4920 generic.go:356] "Generic (PLEG): container finished" podID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerID="1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047" exitCode=0 Oct 30 09:29:29.775229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.775119 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" Oct 30 09:29:29.775229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.775138 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerDied","Data":"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047"} Oct 30 09:29:29.775229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.775180 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk" event={"ID":"5f740194-fd61-4e36-b6b5-e83de6aafa9d","Type":"ContainerDied","Data":"21196b24ba790390431f0d93fc5d6b5b60e0a830398fd0a7eab4b9abeb5cfbce"} Oct 30 09:29:29.785522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.785502 4920 scope.go:122] "RemoveContainer" containerID="263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa" Oct 30 09:29:29.795786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.795764 4920 scope.go:122] "RemoveContainer" containerID="2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120" Oct 30 09:29:29.796190 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:29.796170 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120\": container with ID starting with 2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120 not found: ID does not exist" containerID="2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120" Oct 30 09:29:29.796313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.796202 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120"} err="failed to get container status \"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120\": rpc error: code = NotFound desc = could not find container \"2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120\": container with ID starting with 2900d8a2199a352f2751cf53d923232e3fd01684265e551ea75692873bc73120 not found: ID does not exist" Oct 30 09:29:29.796313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.796251 4920 scope.go:122] "RemoveContainer" containerID="263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa" Oct 30 09:29:29.796502 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:29.796480 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa\": container with ID starting with 263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa not found: ID does not exist" containerID="263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa" Oct 30 09:29:29.796580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.796512 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa"} err="failed to get container status \"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa\": rpc error: code = NotFound desc = could not find container \"263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa\": container with ID starting with 263ff207b691e95bef4d0015a7a6aad046529b88ab6c41fefb50df06ae7426aa not found: ID does not exist" Oct 30 09:29:29.796580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.796533 4920 scope.go:122] "RemoveContainer" containerID="1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047" Oct 30 09:29:29.809681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.809650 4920 scope.go:122] "RemoveContainer" containerID="ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4" Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.930753 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.930819 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.930874 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.930912 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.930984 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.931068 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.931144 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.931178 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.932236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.931372 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\") pod \"bf103a26-1163-4f3d-8d4a-09481a716766\" (UID: \"bf103a26-1163-4f3d-8d4a-09481a716766\") " Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.942997 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.943084 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.945711 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.946006 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.946362 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.954781 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "kube-api-access-dcc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.956867 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.959149 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:29.965235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.962134 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket" pod "bf103a26-1163-4f3d-8d4a-09481a716766" (UID: "bf103a26-1163-4f3d-8d4a-09481a716766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:29.974545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.974281 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:29:29.974545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.974432 4920 scope.go:122] "RemoveContainer" containerID="1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047" Oct 30 09:29:29.976933 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:29.976404 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047\": container with ID starting with 1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047 not found: ID does not exist" containerID="1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047" Oct 30 09:29:29.976933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.976443 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047"} err="failed to get container status \"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047\": rpc error: code = NotFound desc = could not find container \"1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047\": container with ID starting with 1daaa334291147c261e661c78b2ea1aaff234afa0522728c52fcd77c45cc7047 not found: ID does not exist" Oct 30 09:29:29.976933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.976467 4920 scope.go:122] "RemoveContainer" containerID="ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4" Oct 30 09:29:29.978596 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:29.978566 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4\": container with ID starting with ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4 not found: ID does not exist" containerID="ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4" Oct 30 09:29:29.978710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.978605 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4"} err="failed to get container status \"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4\": rpc error: code = NotFound desc = could not find container \"ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4\": container with ID starting with ffa13c21ef232ca59d53c6dbc96ca8830e488dbe1e83c0f5cb6a593e9124d2f4 not found: ID does not exist" Oct 30 09:29:29.979106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:29.979085 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cbd96e-openshift-default-6f77777c49-58frk"] Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033405 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033435 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf103a26-1163-4f3d-8d4a-09481a716766-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033450 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcc5c\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-kube-api-access-dcc5c\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033463 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033474 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033487 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf103a26-1163-4f3d-8d4a-09481a716766-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033500 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033511 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf103a26-1163-4f3d-8d4a-09481a716766-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.033872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.033522 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf103a26-1163-4f3d-8d4a-09481a716766-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:30.120776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.120743 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:29:30.124194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.124172 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9d39c0-openshift-default-bbd66d6c-s9bln"] Oct 30 09:29:30.400510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.400473 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" path="/var/lib/kubelet/pods/5f740194-fd61-4e36-b6b5-e83de6aafa9d/volumes" Oct 30 09:29:30.401213 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.401193 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" path="/var/lib/kubelet/pods/bf103a26-1163-4f3d-8d4a-09481a716766/volumes" Oct 30 09:29:30.489712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:30.489673 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:31.168287 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:31.168238 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:29:32.397370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:32.397271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:34.290746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.290706 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt"] Oct 30 09:29:34.291315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291301 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:34.291315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291316 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291335 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291341 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291359 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291364 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291372 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:34.291417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291377 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:34.291608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291477 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:34.291608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291488 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:34.291608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291496 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:34.291608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.291504 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:34.296343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.296321 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.337249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.332366 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:34.337249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.332488 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:34.343007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.342973 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt"] Oct 30 09:29:34.428256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.397754 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.428579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.428538 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnll\" (UniqueName: \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.428886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.428861 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.440128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.440086 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:34.533349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.533313 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.533717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.533686 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.533870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.533854 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnll\" (UniqueName: \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.535479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.535455 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.540092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.540067 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.556150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.556069 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnll\" (UniqueName: \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\") pod \"maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.645931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.645901 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:34.978156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:34.978111 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt"] Oct 30 09:29:35.850606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:35.850555 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" event={"ID":"45fdfa40-f7d2-4ad3-b079-ae663af267f9","Type":"ContainerStarted","Data":"5e26258d7913e3714bc2e763e319614d0aad3250bb41dfec3baea4de428f63c6"} Oct 30 09:29:35.851088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:35.850630 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" event={"ID":"45fdfa40-f7d2-4ad3-b079-ae663af267f9","Type":"ContainerStarted","Data":"c0b37595763dfae8712b91c0230f403b002a5886534591a3d090d1a7489ae885"} Oct 30 09:29:35.891343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:35.891280 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" podStartSLOduration=1.8912656060000002 podStartE2EDuration="1.891265606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:35.888486406 +0000 UTC m=+2139.611913550" watchObservedRunningTime="2026-10-30 09:29:35.891265606 +0000 UTC m=+2139.614692749" Oct 30 09:29:36.059109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:36.059071 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:36.860364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:36.860325 4920 generic.go:356] "Generic (PLEG): container finished" podID="45fdfa40-f7d2-4ad3-b079-ae663af267f9" containerID="5e26258d7913e3714bc2e763e319614d0aad3250bb41dfec3baea4de428f63c6" exitCode=0 Oct 30 09:29:36.860749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:36.860388 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" event={"ID":"45fdfa40-f7d2-4ad3-b079-ae663af267f9","Type":"ContainerDied","Data":"5e26258d7913e3714bc2e763e319614d0aad3250bb41dfec3baea4de428f63c6"} Oct 30 09:29:37.664525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:37.664464 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:38.077138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.077108 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:38.180193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.180145 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\") pod \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " Oct 30 09:29:38.180193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.180185 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\") pod \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " Oct 30 09:29:38.180479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.180352 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\" (UniqueName: \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\") pod \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\" (UID: \"45fdfa40-f7d2-4ad3-b079-ae663af267f9\") " Oct 30 09:29:38.180708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.180677 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca" pod "45fdfa40-f7d2-4ad3-b079-ae663af267f9" (UID: "45fdfa40-f7d2-4ad3-b079-ae663af267f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:38.182442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.182411 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca" pod "45fdfa40-f7d2-4ad3-b079-ae663af267f9" (UID: "45fdfa40-f7d2-4ad3-b079-ae663af267f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:38.182442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.182427 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll" pod "45fdfa40-f7d2-4ad3-b079-ae663af267f9" (UID: "45fdfa40-f7d2-4ad3-b079-ae663af267f9"). InnerVolumeSpecName "kube-api-access-dsnll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:38.282441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.282318 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:38.282441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.282356 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45fdfa40-f7d2-4ad3-b079-ae663af267f9-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:38.282441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.282371 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsnll\" (UniqueName: \"kubernetes.io/projected/45fdfa40-f7d2-4ad3-b079-ae663af267f9-kube-api-access-dsnll\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:38.880320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.880275 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" event={"ID":"45fdfa40-f7d2-4ad3-b079-ae663af267f9","Type":"ContainerDied","Data":"c0b37595763dfae8712b91c0230f403b002a5886534591a3d090d1a7489ae885"} Oct 30 09:29:38.880531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.880337 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0b37595763dfae8712b91c0230f403b002a5886534591a3d090d1a7489ae885" Oct 30 09:29:38.880531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:38.880347 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt" Oct 30 09:29:39.251729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:39.251648 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:40.767618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.767581 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:40.899425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.899391 4920 generic.go:356] "Generic (PLEG): container finished" podID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerID="77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b" exitCode=0 Oct 30 09:29:40.899598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.899440 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerDied","Data":"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b"} Oct 30 09:29:40.899893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.899869 4920 scope.go:122] "RemoveContainer" containerID="77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b" Oct 30 09:29:40.901431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.901406 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:40.901507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.901472 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:40.901507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:40.901503 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:41.279151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:41.279100 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:29:41.930884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:41.930841 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerStarted","Data":"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f"} Oct 30 09:29:42.273621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.273534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:42.670936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.670879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:42.728369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.728336 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:29:42.729478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.729448 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 09:29:42.729644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.729505 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 09:29:42.784425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.784377 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:42.876965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:42.876916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:43.716201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:43.716158 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:43.728339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:43.728299 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 09:29:43.729032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:43.729000 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 09:29:43.995100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:43.994998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:44.124517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.116818 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:44.462132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.462094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:44.564425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.564395 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:44.670839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.670733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:44.727793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.727681 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 09:29:44.728349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:44.728311 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 09:29:45.122240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.121682 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:45.128653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.128621 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:29:45.140781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.140755 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:29:45.279550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.279517 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:45.395826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.395735 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:45.728844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.728738 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 09:29:45.729387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:45.728846 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 09:29:46.548102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:46.548067 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:46.727314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:46.727274 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 09:29:46.727513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:46.727361 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 09:29:47.732176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:47.732143 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:29:47.732744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:47.732720 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:29:47.733481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:47.733461 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:29:48.075451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:48.075417 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:48.175422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:48.175387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:49.589703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.589663 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:29:49.590542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590520 4920 cpu_manager.go:414] "containerMap: removing container" podUID="45fdfa40-f7d2-4ad3-b079-ae663af267f9" containerName="revoke-keys" Oct 30 09:29:49.590542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590542 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fdfa40-f7d2-4ad3-b079-ae663af267f9" containerName="revoke-keys" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590560 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590566 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f740194-fd61-4e36-b6b5-e83de6aafa9d" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590591 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590596 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590679 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3757924f-64e9-4d9b-a37c-45fe23eb2637" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590690 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf103a26-1163-4f3d-8d4a-09481a716766" containerName="istio-proxy" Oct 30 09:29:49.590725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.590710 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fdfa40-f7d2-4ad3-b079-ae663af267f9" containerName="revoke-keys" Oct 30 09:29:49.595686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.595666 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.617850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.617817 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:29:49.628264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.625163 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:29:49.630391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.629687 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.630391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.630250 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.644005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.643976 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c5447d-openshift-default-dockercfg-nx55s\"" Oct 30 09:29:49.651319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.651291 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:29:49.660140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.660115 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:29:49.665930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.665903 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:29:49.708961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.708927 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.708967 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.708996 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709104 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwnd\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709144 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709203 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709256 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709304 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709347 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709440 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709464 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709498 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709570 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfwh\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709620 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709643 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709713 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709736 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709761 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56c8\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709787 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709882 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709901 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709916 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.709965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.709949 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.710406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.710030 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.710406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.710117 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.710406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.710148 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.710406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.710179 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.806725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.806680 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:49.811153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811120 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.811359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811163 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.811359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811270 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811342 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811382 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.811532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811443 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811497 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfwh\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811574 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.811695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811577 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811644 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.811858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811697 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.811858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811733 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.811858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811768 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56c8\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.811858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811803 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.811858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811847 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811878 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811903 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811956 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.811985 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812047 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812078 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.812131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812109 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812136 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812166 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812269 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwnd\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812303 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812328 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.812521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812328 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.812963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812574 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.812963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812864 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.812963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.812950 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.813205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.813185 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.813615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.813592 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.813708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.813614 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.813854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.813827 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.813918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.813878 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.814090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.814072 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.814169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.814096 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.814444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.814422 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.814526 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.814444 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.815419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.815391 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.815671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.815630 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.816870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.816845 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.818045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.818014 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.819364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.819293 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.819726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.819705 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.819838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.819817 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.832629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.832596 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.834962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.834497 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.834962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.834882 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwnd\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.835798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.835753 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfwh\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.836279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.836259 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56c8\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.837454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.837435 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:49.920910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.920809 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:49.970864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.970836 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:49.980116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:49.980081 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:50.129489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.129420 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" containerID="cri-o://3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682" gracePeriod=25 Oct 30 09:29:50.142119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.142074 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" containerID="cri-o://1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96" gracePeriod=25 Oct 30 09:29:50.248018 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:50.247984 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b41fc0_f9a2_4cb3_bd10_74700220b99c.slice/crio-4fb43df0408a779fc818faf23e7b3a9b46de1f022ceecf6840a4a913ee12f852 WatchSource:0}: Error finding container 4fb43df0408a779fc818faf23e7b3a9b46de1f022ceecf6840a4a913ee12f852: Status 404 returned error can't find the container with id 4fb43df0408a779fc818faf23e7b3a9b46de1f022ceecf6840a4a913ee12f852 Oct 30 09:29:50.249954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.249930 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:29:50.250679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.250648 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.250761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.250715 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.250761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.250745 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.356485 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:50.356444 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c73849_a068_45a9_9ef9_af51b83ea6f7.slice/crio-570af7ab1458620d6182b87d30747935294087765d1d039063851f60e9c48f98 WatchSource:0}: Error finding container 570af7ab1458620d6182b87d30747935294087765d1d039063851f60e9c48f98: Status 404 returned error can't find the container with id 570af7ab1458620d6182b87d30747935294087765d1d039063851f60e9c48f98 Oct 30 09:29:50.358615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.358588 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:29:50.359764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.359731 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.359833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.359794 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.359833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.359825 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.387099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.387077 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:29:50.395700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.395664 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.395812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.395751 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.395812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.395794 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.480196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:50.480138 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:51.061001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.060959 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" event={"ID":"70c73849-a068-45a9-9ef9-af51b83ea6f7","Type":"ContainerStarted","Data":"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670"} Oct 30 09:29:51.061569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.061063 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" event={"ID":"70c73849-a068-45a9-9ef9-af51b83ea6f7","Type":"ContainerStarted","Data":"570af7ab1458620d6182b87d30747935294087765d1d039063851f60e9c48f98"} Oct 30 09:29:51.063027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.062984 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" event={"ID":"8428abf5-dfd4-4a0f-9290-ddce75bb2735","Type":"ContainerStarted","Data":"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d"} Oct 30 09:29:51.063159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.063068 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" event={"ID":"8428abf5-dfd4-4a0f-9290-ddce75bb2735","Type":"ContainerStarted","Data":"afb6a8c78e4cfc74329753965e8a13bd4d6b92639112c79829a62b1a8397eb15"} Oct 30 09:29:51.065036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.065005 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" event={"ID":"f7b41fc0-f9a2-4cb3-bd10-74700220b99c","Type":"ContainerStarted","Data":"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b"} Oct 30 09:29:51.065151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.065046 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" event={"ID":"f7b41fc0-f9a2-4cb3-bd10-74700220b99c","Type":"ContainerStarted","Data":"4fb43df0408a779fc818faf23e7b3a9b46de1f022ceecf6840a4a913ee12f852"} Oct 30 09:29:51.128356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.117981 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podStartSLOduration=2.117960309 podStartE2EDuration="2.117960309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:51.11530425 +0000 UTC m=+2154.838731414" watchObservedRunningTime="2026-10-30 09:29:51.117960309 +0000 UTC m=+2154.841387451" Oct 30 09:29:51.173045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.172989 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podStartSLOduration=2.172976326 podStartE2EDuration="2.172976326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:51.170886478 +0000 UTC m=+2154.894313648" watchObservedRunningTime="2026-10-30 09:29:51.172976326 +0000 UTC m=+2154.896403469" Oct 30 09:29:51.213807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.213740 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podStartSLOduration=2.213718613 podStartE2EDuration="2.213718613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:51.212756884 +0000 UTC m=+2154.936184056" watchObservedRunningTime="2026-10-30 09:29:51.213718613 +0000 UTC m=+2154.937145757" Oct 30 09:29:51.423306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.422372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:51.517576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.517500 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:51.980389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:51.923717 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:52.050292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.000802 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:29:52.050292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.041346 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.076256 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.076323 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.076670 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.076707 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.077172 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.084433 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.090244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.084486 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:52.188430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.144651 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:29:52.188430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.144917 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" containerID="cri-o://1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229" gracePeriod=30 Oct 30 09:29:52.487013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.486631 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:52.696243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.691418 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:29:52.959242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.956429 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.959242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:52.956490 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:53.016242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.008650 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.016242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.008821 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:53.016242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.009296 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.016242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.009351 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:53.690245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.689118 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:29:53.694966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.694936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:53.925242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.924610 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.925242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.924676 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:53.979430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.972711 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.979430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.972790 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:53.983253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.980761 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.983253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:53.980829 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:54.240290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.234067 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:54.559244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.554873 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:54.888539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.873628 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:54.929252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.926782 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:54.929252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.926864 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:54.971785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.971748 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:54.972060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.972033 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:54.982606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.982565 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:54.982870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:54.982840 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:55.416243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.413395 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:29:55.416243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.413458 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:29:55.928239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.925589 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.928239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.925653 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:55.983454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.983414 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.983727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.983700 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:55.984302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.984277 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.984497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:55.984476 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:56.428239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:56.425884 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:56.995243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:56.994872 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:56.995243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:56.994938 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:57.069242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.012865 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.069242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.012928 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:57.069242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.013012 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.069242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.013034 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:57.111872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.111784 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:57.160241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.116067 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp"] Oct 30 09:29:57.290148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.280804 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp"] Oct 30 09:29:57.486487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.290173 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.487129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.424518 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnq6\" (UniqueName: \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.487801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.487720 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.488313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.488293 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.490386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.485811 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:29:57.535430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.527448 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:29:57.539680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.539649 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:29:57.544882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.542855 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:29:57.590830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.590522 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:29:57.613964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.598519 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnq6\" (UniqueName: \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.613964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.598613 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.613964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.598858 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.628495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.616302 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.679692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.679623 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.693278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.690729 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnq6\" (UniqueName: \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\") pod \"maas-api-revoke-sub-c3bc34e8-qwtkp\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.856601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.856507 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:29:57.896243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.896095 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:57.936783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.933325 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.936783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.933391 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 09:29:57.980085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.980006 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.980399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.980351 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:57.980993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.980885 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.980993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:57.980943 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:58.007042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.006939 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:29:58.094951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.094553 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.245822 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.246001 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.246151 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.247693 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.248044 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:58.258314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.248406 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.276456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.268659 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.276456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.268990 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.276456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.269148 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.276456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.274324 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.288738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.288623 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.290842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.290275 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.301699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.301636 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:58.304503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.304369 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:58.315250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.314422 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.337434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.326545 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:58.339440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.339406 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\") pod \"add72d13-1e47-400e-a718-838226d97007\" (UID: \"add72d13-1e47-400e-a718-838226d97007\") " Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342868 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342901 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342916 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add72d13-1e47-400e-a718-838226d97007-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342930 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342945 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342960 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342974 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add72d13-1e47-400e-a718-838226d97007-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.343126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.342986 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add72d13-1e47-400e-a718-838226d97007-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.347612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.347453 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9" pod "add72d13-1e47-400e-a718-838226d97007" (UID: "add72d13-1e47-400e-a718-838226d97007"). InnerVolumeSpecName "kube-api-access-xnfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:58.354199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.354156 4920 generic.go:356] "Generic (PLEG): container finished" podID="add72d13-1e47-400e-a718-838226d97007" containerID="1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229" exitCode=0 Oct 30 09:29:58.354397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.354274 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" Oct 30 09:29:58.354795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.354505 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerDied","Data":"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229"} Oct 30 09:29:58.354795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.354582 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k" event={"ID":"add72d13-1e47-400e-a718-838226d97007","Type":"ContainerDied","Data":"07bda4cd6c17aa31911212e80d91b6ed09885ef27e5b28200532ee2170e8e8cc"} Oct 30 09:29:58.354795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.354619 4920 scope.go:122] "RemoveContainer" containerID="1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229" Oct 30 09:29:58.387999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.387956 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp"] Oct 30 09:29:58.426296 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:29:58.426260 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd15c3a_112f_4868_9ed2_c8749e162237.slice/crio-2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70 WatchSource:0}: Error finding container 2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70: Status 404 returned error can't find the container with id 2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70 Oct 30 09:29:58.441585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.441562 4920 scope.go:122] "RemoveContainer" containerID="4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0" Oct 30 09:29:58.444533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.444493 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfb9\" (UniqueName: \"kubernetes.io/projected/add72d13-1e47-400e-a718-838226d97007-kube-api-access-xnfb9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.512045 4920 scope.go:122] "RemoveContainer" containerID="1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:58.512514 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229\": container with ID starting with 1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229 not found: ID does not exist" containerID="1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.512562 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229"} err="failed to get container status \"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229\": rpc error: code = NotFound desc = could not find container \"1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229\": container with ID starting with 1fc2883f475ccfdea874b0b17d85222d5ca8336c10c2e23b5b5e2ce62cafe229 not found: ID does not exist" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.512587 4920 scope.go:122] "RemoveContainer" containerID="4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:29:58.514118 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0\": container with ID starting with 4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0 not found: ID does not exist" containerID="4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0" Oct 30 09:29:58.523579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.514148 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0"} err="failed to get container status \"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0\": rpc error: code = NotFound desc = could not find container \"4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0\": container with ID starting with 4cf6b4b9488be768b2a48b48a5a575cf8472669211a767f3a96d042534442bc0 not found: ID does not exist" Oct 30 09:29:58.544090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.544056 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:29:58.552569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.552535 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-51e091-openshift-default-6d6d66f5f9-gh45k"] Oct 30 09:29:58.971901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.971856 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 09:29:58.972230 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.972167 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 09:29:58.981198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.981164 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 09:29:58.981471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:58.981425 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 09:29:59.369657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.369620 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" event={"ID":"bdd15c3a-112f-4868-9ed2-c8749e162237","Type":"ContainerStarted","Data":"7299f881d14e778c98c896809140e9bcbb3013dddfe8903d6a2cad0558f7e705"} Oct 30 09:29:59.370387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.370356 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" event={"ID":"bdd15c3a-112f-4868-9ed2-c8749e162237","Type":"ContainerStarted","Data":"2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70"} Oct 30 09:29:59.416952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.416886 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" podStartSLOduration=3.416866502 podStartE2EDuration="3.416866502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:59.415524305 +0000 UTC m=+2163.138951449" watchObservedRunningTime="2026-10-30 09:29:59.416866502 +0000 UTC m=+2163.140293647" Oct 30 09:29:59.779547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.779511 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:59.781142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.781114 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:29:59.782307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:29:59.782274 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:30:00.087066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.087005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:00.147133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.147092 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:00.149117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.149088 4920 cpu_manager.go:414] "containerMap: removing container" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.149325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.149310 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.149667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.149651 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.149790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.149779 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.154262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.154238 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.200196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.200073 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:00.218950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.218910 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:00.223864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.223830 4920 cpu_manager.go:414] "containerMap: removing container" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.224260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.224240 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add72d13-1e47-400e-a718-838226d97007" containerName="istio-proxy" Oct 30 09:30:00.244345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.243341 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:00.251309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.251253 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcft6\" (UniqueName: \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.251506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.251337 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.254062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.254040 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:30:00.254358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.254289 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:00.254545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.254117 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.264985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.264947 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.265341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.264862 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:30:00.265529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.265512 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:00.265705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.265690 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:30:00.265848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.265834 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:00.354836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.354745 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcft6\" (UniqueName: \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.354836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.354807 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.355102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.354883 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj2q\" (UniqueName: \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.355102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.354979 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.356466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.356435 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.382959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.378298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcft6\" (UniqueName: \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.397026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.396987 4920 generic.go:356] "Generic (PLEG): container finished" podID="bdd15c3a-112f-4868-9ed2-c8749e162237" containerID="7299f881d14e778c98c896809140e9bcbb3013dddfe8903d6a2cad0558f7e705" exitCode=0 Oct 30 09:30:00.413781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.413126 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add72d13-1e47-400e-a718-838226d97007" path="/var/lib/kubelet/pods/add72d13-1e47-400e-a718-838226d97007/volumes" Oct 30 09:30:00.413781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.413658 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" event={"ID":"bdd15c3a-112f-4868-9ed2-c8749e162237","Type":"ContainerDied","Data":"7299f881d14e778c98c896809140e9bcbb3013dddfe8903d6a2cad0558f7e705"} Oct 30 09:30:00.457278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.456489 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.457278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.456676 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4jf\" (UniqueName: \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.457278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.456780 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.457278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.456823 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj2q\" (UniqueName: \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.459403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.459376 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.529309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.526863 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj2q\" (UniqueName: \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\") pod \"maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.556522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.556488 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:00.614105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.613768 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:00.685327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.617195 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4jf\" (UniqueName: \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.685327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.617355 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.685327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.618197 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.685327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.660999 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4jf\" (UniqueName: \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\") pod \"maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.915854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.910470 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:00.982347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.981787 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:30:00.982347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:00.981849 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:01.210028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:01.209984 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:01.501354 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:01.495738 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:30:01.659258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:01.659194 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:01.781574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:01.757803 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerStarted","Data":"688dd5ab9fd5d0c3fce90f96c1e9d30657277873df859b34312dae0862413e32"} Oct 30 09:30:01.783029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:01.782946 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:02.039130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.036363 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:30:02.039130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.036423 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:02.076715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.073679 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:02.144058 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:02.144029 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf883161b_2344_4426_9fd0_b479d0c98bc2.slice/crio-1ed30072153f497d475ad1d66e6c529222ab196644d600a545069cea06efeb19 WatchSource:0}: Error finding container 1ed30072153f497d475ad1d66e6c529222ab196644d600a545069cea06efeb19: Status 404 returned error can't find the container with id 1ed30072153f497d475ad1d66e6c529222ab196644d600a545069cea06efeb19 Oct 30 09:30:02.352529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.352493 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:30:02.358836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.358767 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:30:02.359987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.359960 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:30:02.379170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.379136 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:30:02.430845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.430815 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\" (UniqueName: \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\") pod \"bdd15c3a-112f-4868-9ed2-c8749e162237\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " Oct 30 09:30:02.431176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.431155 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\") pod \"bdd15c3a-112f-4868-9ed2-c8749e162237\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " Oct 30 09:30:02.431333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.431311 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\") pod \"bdd15c3a-112f-4868-9ed2-c8749e162237\" (UID: \"bdd15c3a-112f-4868-9ed2-c8749e162237\") " Oct 30 09:30:02.431619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.431579 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca" pod "bdd15c3a-112f-4868-9ed2-c8749e162237" (UID: "bdd15c3a-112f-4868-9ed2-c8749e162237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:02.432471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.432261 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.501804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.501768 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6" pod "bdd15c3a-112f-4868-9ed2-c8749e162237" (UID: "bdd15c3a-112f-4868-9ed2-c8749e162237"). InnerVolumeSpecName "kube-api-access-cwnq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:02.503326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.503297 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca" pod "bdd15c3a-112f-4868-9ed2-c8749e162237" (UID: "bdd15c3a-112f-4868-9ed2-c8749e162237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:02.538419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.538361 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwnq6\" (UniqueName: \"kubernetes.io/projected/bdd15c3a-112f-4868-9ed2-c8749e162237-kube-api-access-cwnq6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.538665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.538643 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd15c3a-112f-4868-9ed2-c8749e162237-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.664167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.664124 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:30:02.767823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.767784 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerStarted","Data":"1ed30072153f497d475ad1d66e6c529222ab196644d600a545069cea06efeb19"} Oct 30 09:30:02.769423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.769395 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerStarted","Data":"191cd99f99b3bc169217271c260d26f31baabb14f0e638794e6e7324bd228de8"} Oct 30 09:30:02.771082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.771055 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerStarted","Data":"edff33ce8151934a4f724bb1fe4c2c0ae92e2ffe84c780b90237731fb31fe60a"} Oct 30 09:30:02.771207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.771101 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerStarted","Data":"23a1d2c120670785861c3a747d0603d41429c9db423c06b34bb3ad16d461b651"} Oct 30 09:30:02.772772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.772751 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" Oct 30 09:30:02.772870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.772783 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp" event={"ID":"bdd15c3a-112f-4868-9ed2-c8749e162237","Type":"ContainerDied","Data":"2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70"} Oct 30 09:30:02.772870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.772832 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf77190b3bd55aa21f07931ea7df9ebca9ffb74ff11394ed39939f027218e70" Oct 30 09:30:02.813314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.813255 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" podStartSLOduration=2.813233064 podStartE2EDuration="2.813233064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:02.808961905 +0000 UTC m=+2166.532389064" watchObservedRunningTime="2026-10-30 09:30:02.813233064 +0000 UTC m=+2166.536660259" Oct 30 09:30:02.861768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.861705 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" podStartSLOduration=2.86168521 podStartE2EDuration="2.86168521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:02.860099496 +0000 UTC m=+2166.583526630" watchObservedRunningTime="2026-10-30 09:30:02.86168521 +0000 UTC m=+2166.585112353" Oct 30 09:30:02.917594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.917492 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx"] Oct 30 09:30:02.918507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.918479 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bdd15c3a-112f-4868-9ed2-c8749e162237" containerName="revoke-keys" Oct 30 09:30:02.918507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.918508 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd15c3a-112f-4868-9ed2-c8749e162237" containerName="revoke-keys" Oct 30 09:30:02.918732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.918659 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd15c3a-112f-4868-9ed2-c8749e162237" containerName="revoke-keys" Oct 30 09:30:02.923024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.923002 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:02.937475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.937146 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:02.944992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:02.944955 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx"] Oct 30 09:30:03.045793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.045749 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.046006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.045952 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dg8\" (UniqueName: \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.046143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.046123 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.147490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.147460 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.147648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.147557 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dg8\" (UniqueName: \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.147703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.147646 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.148243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.148209 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.149951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.149926 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.165949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.165917 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dg8\" (UniqueName: \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\") pod \"maas-api-revoke-sub-39b03bff-5kmpx\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.241904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.241793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:03.251948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.251910 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:03.574324 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:03.574293 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8f1221_674b_4993_8b67_cfe5ea6d9177.slice/crio-db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26 WatchSource:0}: Error finding container db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26: Status 404 returned error can't find the container with id db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26 Oct 30 09:30:03.575680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.575655 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx"] Oct 30 09:30:03.674537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.674497 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:30:03.783798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.783763 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" event={"ID":"aa8f1221-674b-4993-8b67-cfe5ea6d9177","Type":"ContainerStarted","Data":"db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26"} Oct 30 09:30:03.785466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.785438 4920 generic.go:356] "Generic (PLEG): container finished" podID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerID="39262478943bdc34eee969b19374450409baa338d8cba1cdd9a2097257d07e1a" exitCode=60 Oct 30 09:30:03.785590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.785515 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerDied","Data":"39262478943bdc34eee969b19374450409baa338d8cba1cdd9a2097257d07e1a"} Oct 30 09:30:03.785819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.785805 4920 scope.go:122] "RemoveContainer" containerID="39262478943bdc34eee969b19374450409baa338d8cba1cdd9a2097257d07e1a" Oct 30 09:30:03.787339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.787319 4920 generic.go:356] "Generic (PLEG): container finished" podID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerID="191cd99f99b3bc169217271c260d26f31baabb14f0e638794e6e7324bd228de8" exitCode=60 Oct 30 09:30:03.787454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.787412 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerDied","Data":"191cd99f99b3bc169217271c260d26f31baabb14f0e638794e6e7324bd228de8"} Oct 30 09:30:03.787658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.787640 4920 scope.go:122] "RemoveContainer" containerID="191cd99f99b3bc169217271c260d26f31baabb14f0e638794e6e7324bd228de8" Oct 30 09:30:03.789060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.789038 4920 generic.go:356] "Generic (PLEG): container finished" podID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerID="edff33ce8151934a4f724bb1fe4c2c0ae92e2ffe84c780b90237731fb31fe60a" exitCode=60 Oct 30 09:30:03.789177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.789105 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerDied","Data":"edff33ce8151934a4f724bb1fe4c2c0ae92e2ffe84c780b90237731fb31fe60a"} Oct 30 09:30:03.789523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:03.789326 4920 scope.go:122] "RemoveContainer" containerID="edff33ce8151934a4f724bb1fe4c2c0ae92e2ffe84c780b90237731fb31fe60a" Oct 30 09:30:04.801373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.801324 4920 generic.go:356] "Generic (PLEG): container finished" podID="aa8f1221-674b-4993-8b67-cfe5ea6d9177" containerID="7b75ab852e75bfd777048672ebba4057b89140e7f3bb1d4a4af9d2949c7f228d" exitCode=0 Oct 30 09:30:04.801912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.801838 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" event={"ID":"aa8f1221-674b-4993-8b67-cfe5ea6d9177","Type":"ContainerDied","Data":"7b75ab852e75bfd777048672ebba4057b89140e7f3bb1d4a4af9d2949c7f228d"} Oct 30 09:30:04.804890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.804858 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerStarted","Data":"bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c"} Oct 30 09:30:04.807583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.807556 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerStarted","Data":"5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3"} Oct 30 09:30:04.809646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.809620 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerStarted","Data":"ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea"} Oct 30 09:30:04.925577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.925542 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:04.952768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:04.952714 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" podStartSLOduration=4.952701135 podStartE2EDuration="4.952701135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:04.951583388 +0000 UTC m=+2168.675010532" watchObservedRunningTime="2026-10-30 09:30:04.952701135 +0000 UTC m=+2168.676128277" Oct 30 09:30:05.821339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.821299 4920 generic.go:356] "Generic (PLEG): container finished" podID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerID="ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea" exitCode=60 Oct 30 09:30:05.821894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.821344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerDied","Data":"ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea"} Oct 30 09:30:05.821894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.821418 4920 scope.go:122] "RemoveContainer" containerID="edff33ce8151934a4f724bb1fe4c2c0ae92e2ffe84c780b90237731fb31fe60a" Oct 30 09:30:05.821894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.821737 4920 scope.go:122] "RemoveContainer" containerID="ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea" Oct 30 09:30:05.822154 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:05.822129 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc_odh-ai-gateway-infra(6dc41e00-d519-45b2-868c-edfff1b0449f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" Oct 30 09:30:05.823877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.823856 4920 generic.go:356] "Generic (PLEG): container finished" podID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerID="bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c" exitCode=60 Oct 30 09:30:05.823985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.823946 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerDied","Data":"bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c"} Oct 30 09:30:05.824262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.824247 4920 scope.go:122] "RemoveContainer" containerID="bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c" Oct 30 09:30:05.824597 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:05.824578 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz_odh-ai-gateway-infra(f883161b-2344-4426-9fd0-b479d0c98bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" Oct 30 09:30:05.826140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.826118 4920 generic.go:356] "Generic (PLEG): container finished" podID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerID="5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3" exitCode=60 Oct 30 09:30:05.826260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.826210 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerDied","Data":"5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3"} Oct 30 09:30:05.826536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.826515 4920 scope.go:122] "RemoveContainer" containerID="5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3" Oct 30 09:30:05.826876 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:05.826845 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r_odh-ai-gateway-infra(ca164d4e-9296-4e9a-8b86-40e773270ce6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" Oct 30 09:30:05.837616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.837585 4920 scope.go:122] "RemoveContainer" containerID="39262478943bdc34eee969b19374450409baa338d8cba1cdd9a2097257d07e1a" Oct 30 09:30:05.854199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:05.854174 4920 scope.go:122] "RemoveContainer" containerID="191cd99f99b3bc169217271c260d26f31baabb14f0e638794e6e7324bd228de8" Oct 30 09:30:06.009329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.009295 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:06.099653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.099566 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\") pod \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " Oct 30 09:30:06.099653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.099621 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\" (UniqueName: \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\") pod \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " Oct 30 09:30:06.099886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.099816 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\") pod \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\" (UID: \"aa8f1221-674b-4993-8b67-cfe5ea6d9177\") " Oct 30 09:30:06.100422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.100388 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca" pod "aa8f1221-674b-4993-8b67-cfe5ea6d9177" (UID: "aa8f1221-674b-4993-8b67-cfe5ea6d9177"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:06.102234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.102201 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8" pod "aa8f1221-674b-4993-8b67-cfe5ea6d9177" (UID: "aa8f1221-674b-4993-8b67-cfe5ea6d9177"). InnerVolumeSpecName "kube-api-access-q8dg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:06.102713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.102680 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca" pod "aa8f1221-674b-4993-8b67-cfe5ea6d9177" (UID: "aa8f1221-674b-4993-8b67-cfe5ea6d9177"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:06.201981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.201942 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:06.201981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.201975 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa8f1221-674b-4993-8b67-cfe5ea6d9177-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:06.201981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.201986 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8dg8\" (UniqueName: \"kubernetes.io/projected/aa8f1221-674b-4993-8b67-cfe5ea6d9177-kube-api-access-q8dg8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:06.582344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.582307 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:06.850998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.850969 4920 scope.go:122] "RemoveContainer" containerID="ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea" Oct 30 09:30:06.851436 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:06.851336 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc_odh-ai-gateway-infra(6dc41e00-d519-45b2-868c-edfff1b0449f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" Oct 30 09:30:06.852164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.852138 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" Oct 30 09:30:06.852292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.852171 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx" event={"ID":"aa8f1221-674b-4993-8b67-cfe5ea6d9177","Type":"ContainerDied","Data":"db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26"} Oct 30 09:30:06.852292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.852273 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db71a939f3dfd6d6b623c83e6e74dbe2660ab64be69b3a36d3634ad067a79e26" Oct 30 09:30:06.853999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.853977 4920 scope.go:122] "RemoveContainer" containerID="bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c" Oct 30 09:30:06.854455 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:06.854276 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz_odh-ai-gateway-infra(f883161b-2344-4426-9fd0-b479d0c98bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" Oct 30 09:30:06.855812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:06.855796 4920 scope.go:122] "RemoveContainer" containerID="5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3" Oct 30 09:30:06.856007 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:06.855990 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r_odh-ai-gateway-infra(ca164d4e-9296-4e9a-8b86-40e773270ce6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" Oct 30 09:30:08.259485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:08.259448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:09.371919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.371888 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n"] Oct 30 09:30:09.372601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.372580 4920 cpu_manager.go:414] "containerMap: removing container" podUID="aa8f1221-674b-4993-8b67-cfe5ea6d9177" containerName="revoke-keys" Oct 30 09:30:09.372601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.372600 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8f1221-674b-4993-8b67-cfe5ea6d9177" containerName="revoke-keys" Oct 30 09:30:09.372829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.372816 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8f1221-674b-4993-8b67-cfe5ea6d9177" containerName="revoke-keys" Oct 30 09:30:09.376517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.376498 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.390434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.390400 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:09.394125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.394099 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n"] Oct 30 09:30:09.549783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.549751 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.549969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.549798 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.549969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.549874 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5p6w\" (UniqueName: \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.651423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.651340 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.651423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.651401 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.651646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.651424 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5p6w\" (UniqueName: \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.652093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.652068 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.653872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.653849 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.671558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.671521 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5p6w\" (UniqueName: \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\") pod \"maas-api-revoke-sub-552eca70-mjs9n\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.694980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.694946 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:09.914413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:09.914322 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:10.032145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.032114 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n"] Oct 30 09:30:10.441292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.441265 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:30:10.464143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.464121 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:30:10.466853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.466827 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\") pod \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " Oct 30 09:30:10.466936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.466913 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\") pod \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " Oct 30 09:30:10.466999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.466986 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\" (UniqueName: \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\") pod \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\" (UID: \"4dd4f29d-c23f-434b-a52d-c30df3f9618e\") " Oct 30 09:30:10.467043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.467036 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\" (UniqueName: \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\") pod \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\" (UID: \"0538287e-480f-4c4b-98f8-ce86b0b0fa43\") " Oct 30 09:30:10.467301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.467277 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume" pod "4dd4f29d-c23f-434b-a52d-c30df3f9618e" (UID: "4dd4f29d-c23f-434b-a52d-c30df3f9618e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:10.467396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.467292 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume" pod "0538287e-480f-4c4b-98f8-ce86b0b0fa43" (UID: "0538287e-480f-4c4b-98f8-ce86b0b0fa43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:10.467525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.467504 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0538287e-480f-4c4b-98f8-ce86b0b0fa43-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:10.467587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.467531 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd4f29d-c23f-434b-a52d-c30df3f9618e-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:10.469055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.469035 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc" pod "4dd4f29d-c23f-434b-a52d-c30df3f9618e" (UID: "4dd4f29d-c23f-434b-a52d-c30df3f9618e"). InnerVolumeSpecName "kube-api-access-bvcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:10.469141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.469116 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm" pod "0538287e-480f-4c4b-98f8-ce86b0b0fa43" (UID: "0538287e-480f-4c4b-98f8-ce86b0b0fa43"). InnerVolumeSpecName "kube-api-access-b68pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:10.568338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.568300 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcwc\" (UniqueName: \"kubernetes.io/projected/4dd4f29d-c23f-434b-a52d-c30df3f9618e-kube-api-access-bvcwc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:10.568338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.568331 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68pm\" (UniqueName: \"kubernetes.io/projected/0538287e-480f-4c4b-98f8-ce86b0b0fa43-kube-api-access-b68pm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:10.650452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.650420 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:10.750827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.750736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:10.890183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.890126 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" event={"ID":"f8f22005-cec9-45c0-a3a8-674d16e0bce3","Type":"ContainerStarted","Data":"64ab62cb4d6103deab3d831e1d19acdc4692163bb0950bb83ae7e8513e1bcc2b"} Oct 30 09:30:10.890363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.890256 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" event={"ID":"f8f22005-cec9-45c0-a3a8-674d16e0bce3","Type":"ContainerStarted","Data":"e8fb68f433ac28883d795eb7fa58d9199791a7dd98c0ec8289d1ce38b7189a6d"} Oct 30 09:30:10.891542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.891513 4920 generic.go:356] "Generic (PLEG): container finished" podID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerID="1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96" exitCode=0 Oct 30 09:30:10.891650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.891556 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" Oct 30 09:30:10.891650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.891594 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" event={"ID":"4dd4f29d-c23f-434b-a52d-c30df3f9618e","Type":"ContainerDied","Data":"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96"} Oct 30 09:30:10.891650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.891640 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6" event={"ID":"4dd4f29d-c23f-434b-a52d-c30df3f9618e","Type":"ContainerDied","Data":"e25b46a77cd2fa3083a7265630746169450d7ceeb4966d481d816137a17dbef1"} Oct 30 09:30:10.891785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.891672 4920 scope.go:122] "RemoveContainer" containerID="1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96" Oct 30 09:30:10.893180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.893158 4920 generic.go:356] "Generic (PLEG): container finished" podID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerID="3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682" exitCode=0 Oct 30 09:30:10.893286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.893242 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" Oct 30 09:30:10.893286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.893255 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" event={"ID":"0538287e-480f-4c4b-98f8-ce86b0b0fa43","Type":"ContainerDied","Data":"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682"} Oct 30 09:30:10.893411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.893318 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq" event={"ID":"0538287e-480f-4c4b-98f8-ce86b0b0fa43","Type":"ContainerDied","Data":"a480d3f981afef42e89549a4c2924193c4307f58c764254e00b54b300f1aabee"} Oct 30 09:30:10.905478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.905453 4920 scope.go:122] "RemoveContainer" containerID="1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96" Oct 30 09:30:10.905865 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:10.905835 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96\": container with ID starting with 1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96 not found: ID does not exist" containerID="1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96" Oct 30 09:30:10.905963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.905876 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96"} err="failed to get container status \"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96\": rpc error: code = NotFound desc = could not find container \"1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96\": container with ID starting with 1fc88bd1d4ba06d56778176de0197ac01a5e4acb0a870d5a08fee62a400d1b96 not found: ID does not exist" Oct 30 09:30:10.905963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.905901 4920 scope.go:122] "RemoveContainer" containerID="3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682" Oct 30 09:30:10.918071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.918046 4920 scope.go:122] "RemoveContainer" containerID="3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682" Oct 30 09:30:10.918504 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:10.918480 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682\": container with ID starting with 3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682 not found: ID does not exist" containerID="3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682" Oct 30 09:30:10.918578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.918512 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682"} err="failed to get container status \"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682\": rpc error: code = NotFound desc = could not find container \"3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682\": container with ID starting with 3ab05f3ba1907cde537362532af7b5add9218720541fc09462a79a0ac0885682 not found: ID does not exist" Oct 30 09:30:10.934147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.934096 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" podStartSLOduration=1.934082517 podStartE2EDuration="1.934082517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:10.930269069 +0000 UTC m=+2174.653696231" watchObservedRunningTime="2026-10-30 09:30:10.934082517 +0000 UTC m=+2174.657509693" Oct 30 09:30:10.967798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.967768 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:30:10.971492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:10.971462 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c9232f-7949fb67d-bb8hq"] Oct 30 09:30:11.005829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.005743 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:30:11.011520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.011490 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c9232f-59b86dfc9f-wngf6"] Oct 30 09:30:11.690821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.690773 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:30:11.691256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.690997 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" containerID="cri-o://b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e" gracePeriod=30 Oct 30 09:30:11.903232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.903192 4920 generic.go:356] "Generic (PLEG): container finished" podID="f8f22005-cec9-45c0-a3a8-674d16e0bce3" containerID="64ab62cb4d6103deab3d831e1d19acdc4692163bb0950bb83ae7e8513e1bcc2b" exitCode=0 Oct 30 09:30:11.903429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.903250 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" event={"ID":"f8f22005-cec9-45c0-a3a8-674d16e0bce3","Type":"ContainerDied","Data":"64ab62cb4d6103deab3d831e1d19acdc4692163bb0950bb83ae7e8513e1bcc2b"} Oct 30 09:30:11.966509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:11.966424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:12.398945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:12.398912 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" path="/var/lib/kubelet/pods/0538287e-480f-4c4b-98f8-ce86b0b0fa43/volumes" Oct 30 09:30:12.399536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:12.399494 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" path="/var/lib/kubelet/pods/4dd4f29d-c23f-434b-a52d-c30df3f9618e/volumes" Oct 30 09:30:13.095134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.095109 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:13.201744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.201650 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\" (UniqueName: \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\") pod \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " Oct 30 09:30:13.201915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.201756 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\") pod \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " Oct 30 09:30:13.202009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.201971 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\") pod \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\" (UID: \"f8f22005-cec9-45c0-a3a8-674d16e0bce3\") " Oct 30 09:30:13.202321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.202294 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca" pod "f8f22005-cec9-45c0-a3a8-674d16e0bce3" (UID: "f8f22005-cec9-45c0-a3a8-674d16e0bce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:13.202568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.202549 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:13.203988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.203963 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca" pod "f8f22005-cec9-45c0-a3a8-674d16e0bce3" (UID: "f8f22005-cec9-45c0-a3a8-674d16e0bce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:13.204334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.204312 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w" pod "f8f22005-cec9-45c0-a3a8-674d16e0bce3" (UID: "f8f22005-cec9-45c0-a3a8-674d16e0bce3"). InnerVolumeSpecName "kube-api-access-d5p6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:13.304305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.304270 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5p6w\" (UniqueName: \"kubernetes.io/projected/f8f22005-cec9-45c0-a3a8-674d16e0bce3-kube-api-access-d5p6w\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:13.304305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.304300 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f22005-cec9-45c0-a3a8-674d16e0bce3-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:13.552576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.552542 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:13.924072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.923985 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" event={"ID":"f8f22005-cec9-45c0-a3a8-674d16e0bce3","Type":"ContainerDied","Data":"e8fb68f433ac28883d795eb7fa58d9199791a7dd98c0ec8289d1ce38b7189a6d"} Oct 30 09:30:13.924072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.924047 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8fb68f433ac28883d795eb7fa58d9199791a7dd98c0ec8289d1ce38b7189a6d" Oct 30 09:30:13.924072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:13.923993 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n" Oct 30 09:30:15.520232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:15.520182 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:15.620470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:15.620429 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:15.720430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:15.720395 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:15.819921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:15.819887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:16.395775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.395746 4920 scope.go:122] "RemoveContainer" containerID="bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c" Oct 30 09:30:16.991587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.991559 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:30:16.997326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.997294 4920 generic.go:356] "Generic (PLEG): container finished" podID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerID="b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e" exitCode=0 Oct 30 09:30:16.997518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.997342 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" Oct 30 09:30:16.997518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.997390 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" event={"ID":"4e969a6e-ab0f-49c0-bd19-9c8670209bdc","Type":"ContainerDied","Data":"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e"} Oct 30 09:30:16.997518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.997462 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8" event={"ID":"4e969a6e-ab0f-49c0-bd19-9c8670209bdc","Type":"ContainerDied","Data":"2b85cf5321716ee9fa62eac7658ae831d49b9e03b6a085c70741080ca2bcaa82"} Oct 30 09:30:16.997518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.997496 4920 scope.go:122] "RemoveContainer" containerID="b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e" Oct 30 09:30:16.999863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:16.999836 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerStarted","Data":"593e1d614c4c0ef5949ab324d6aee2a5a1f7e95ec4048faf867b964707d3f3e7"} Oct 30 09:30:17.011510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.011487 4920 scope.go:122] "RemoveContainer" containerID="b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e" Oct 30 09:30:17.011876 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:17.011845 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e\": container with ID starting with b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e not found: ID does not exist" containerID="b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e" Oct 30 09:30:17.011978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.011888 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e"} err="failed to get container status \"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e\": rpc error: code = NotFound desc = could not find container \"b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e\": container with ID starting with b6f10c459d9feea2fd959ed03306b5e39642609a809e02aeb844b1ab9357578e not found: ID does not exist" Oct 30 09:30:17.059815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.059783 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.059853 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.059905 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.059939 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.059993 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060112 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060185 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060265 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060320 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\") pod \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\" (UID: \"4e969a6e-ab0f-49c0-bd19-9c8670209bdc\") " Oct 30 09:30:17.060617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060106 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:17.061116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.061070 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.061116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.060123 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:17.061316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.061248 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:17.061518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.061495 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:17.062331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.062302 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:17.104275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.097781 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:30:17.104275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.097819 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:17.104275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.097845 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "kube-api-access-9ngg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:17.104575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.104509 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy" pod "4e969a6e-ab0f-49c0-bd19-9c8670209bdc" (UID: "4e969a6e-ab0f-49c0-bd19-9c8670209bdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:17.161132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.161081 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:17.161990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.161966 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.161997 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162008 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162017 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162025 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162032 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162040 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.162070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.162049 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngg5\" (UniqueName: \"kubernetes.io/projected/4e969a6e-ab0f-49c0-bd19-9c8670209bdc-kube-api-access-9ngg5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:17.348312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.348281 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:30:17.359827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:17.359793 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-mf8c8"] Oct 30 09:30:18.012876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.012842 4920 generic.go:356] "Generic (PLEG): container finished" podID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerID="593e1d614c4c0ef5949ab324d6aee2a5a1f7e95ec4048faf867b964707d3f3e7" exitCode=60 Oct 30 09:30:18.013392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.012914 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerDied","Data":"593e1d614c4c0ef5949ab324d6aee2a5a1f7e95ec4048faf867b964707d3f3e7"} Oct 30 09:30:18.013392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.013023 4920 scope.go:122] "RemoveContainer" containerID="bb30d5183f5b82dc5ee5c03770c07233a07d322f91c2c28ae126c8fed616a06c" Oct 30 09:30:18.013506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.013470 4920 scope.go:122] "RemoveContainer" containerID="593e1d614c4c0ef5949ab324d6aee2a5a1f7e95ec4048faf867b964707d3f3e7" Oct 30 09:30:18.013858 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:18.013836 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz_odh-ai-gateway-infra(f883161b-2344-4426-9fd0-b479d0c98bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" Oct 30 09:30:18.097417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.097381 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:18.390924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.390893 4920 scope.go:122] "RemoveContainer" containerID="ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea" Oct 30 09:30:18.399626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.399601 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" path="/var/lib/kubelet/pods/4e969a6e-ab0f-49c0-bd19-9c8670209bdc/volumes" Oct 30 09:30:18.678562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:18.678479 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:19.028714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.028678 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerStarted","Data":"0875edf7d720c98e0f937cfb6f2af1ac24a109ddef6a3e392587cc490c8d93c9"} Oct 30 09:30:19.210484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.210452 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:19.299036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.299001 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\") pod \"f883161b-2344-4426-9fd0-b479d0c98bc2\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " Oct 30 09:30:19.299247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.299189 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\" (UniqueName: \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\") pod \"f883161b-2344-4426-9fd0-b479d0c98bc2\" (UID: \"f883161b-2344-4426-9fd0-b479d0c98bc2\") " Oct 30 09:30:19.299558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.299532 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca" pod "f883161b-2344-4426-9fd0-b479d0c98bc2" (UID: "f883161b-2344-4426-9fd0-b479d0c98bc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:19.299834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.299808 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f883161b-2344-4426-9fd0-b479d0c98bc2-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:19.302016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.301970 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf" pod "f883161b-2344-4426-9fd0-b479d0c98bc2" (UID: "f883161b-2344-4426-9fd0-b479d0c98bc2"). InnerVolumeSpecName "kube-api-access-zm4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:19.391235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.391178 4920 scope.go:122] "RemoveContainer" containerID="5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3" Oct 30 09:30:19.401247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:19.401199 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4jf\" (UniqueName: \"kubernetes.io/projected/f883161b-2344-4426-9fd0-b479d0c98bc2-kube-api-access-zm4jf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:20.040524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.040488 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerStarted","Data":"0df094e118389a11057c5da81c0f674049640eac95fc83fa04a0b8e2715ab7aa"} Oct 30 09:30:20.042505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.042472 4920 generic.go:356] "Generic (PLEG): container finished" podID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerID="0875edf7d720c98e0f937cfb6f2af1ac24a109ddef6a3e392587cc490c8d93c9" exitCode=60 Oct 30 09:30:20.042658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.042542 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerDied","Data":"0875edf7d720c98e0f937cfb6f2af1ac24a109ddef6a3e392587cc490c8d93c9"} Oct 30 09:30:20.042658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.042605 4920 scope.go:122] "RemoveContainer" containerID="ad063775ddfb150d13b45a1becb52856a681fda0bbe7380270ab6928b8f50bea" Oct 30 09:30:20.042993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.042971 4920 scope.go:122] "RemoveContainer" containerID="0875edf7d720c98e0f937cfb6f2af1ac24a109ddef6a3e392587cc490c8d93c9" Oct 30 09:30:20.043404 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:20.043370 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc_odh-ai-gateway-infra(6dc41e00-d519-45b2-868c-edfff1b0449f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" Oct 30 09:30:20.044419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.044399 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" event={"ID":"f883161b-2344-4426-9fd0-b479d0c98bc2","Type":"ContainerDied","Data":"1ed30072153f497d475ad1d66e6c529222ab196644d600a545069cea06efeb19"} Oct 30 09:30:20.044518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.044471 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz" Oct 30 09:30:20.056505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.056483 4920 scope.go:122] "RemoveContainer" containerID="593e1d614c4c0ef5949ab324d6aee2a5a1f7e95ec4048faf867b964707d3f3e7" Oct 30 09:30:20.112993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.112954 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:20.117157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.117125 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:20.122050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.122024 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d87c85-29889210-tkpfz"] Oct 30 09:30:20.312264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.312162 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:20.420481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:20.420066 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" path="/var/lib/kubelet/pods/f883161b-2344-4426-9fd0-b479d0c98bc2/volumes" Oct 30 09:30:21.057977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.057949 4920 generic.go:356] "Generic (PLEG): container finished" podID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerID="0df094e118389a11057c5da81c0f674049640eac95fc83fa04a0b8e2715ab7aa" exitCode=60 Oct 30 09:30:21.058417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.058013 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerDied","Data":"0df094e118389a11057c5da81c0f674049640eac95fc83fa04a0b8e2715ab7aa"} Oct 30 09:30:21.058417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.058056 4920 scope.go:122] "RemoveContainer" containerID="5b8240a5262d32963d49fe1725db462b5eea44e4bdbdbbc4d7032f7051da88c3" Oct 30 09:30:21.058551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.058433 4920 scope.go:122] "RemoveContainer" containerID="0df094e118389a11057c5da81c0f674049640eac95fc83fa04a0b8e2715ab7aa" Oct 30 09:30:21.058802 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:21.058782 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r_odh-ai-gateway-infra(ca164d4e-9296-4e9a-8b86-40e773270ce6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" Oct 30 09:30:21.090810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.090772 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:21.243069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.243041 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:21.333626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.333517 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\" (UniqueName: \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\") pod \"6dc41e00-d519-45b2-868c-edfff1b0449f\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " Oct 30 09:30:21.333805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.333773 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\") pod \"6dc41e00-d519-45b2-868c-edfff1b0449f\" (UID: \"6dc41e00-d519-45b2-868c-edfff1b0449f\") " Oct 30 09:30:21.334346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.334313 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca" pod "6dc41e00-d519-45b2-868c-edfff1b0449f" (UID: "6dc41e00-d519-45b2-868c-edfff1b0449f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:21.336980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.336942 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q" pod "6dc41e00-d519-45b2-868c-edfff1b0449f" (UID: "6dc41e00-d519-45b2-868c-edfff1b0449f"). InnerVolumeSpecName "kube-api-access-vsj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:21.436417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.436387 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsj2q\" (UniqueName: \"kubernetes.io/projected/6dc41e00-d519-45b2-868c-edfff1b0449f-kube-api-access-vsj2q\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:21.436633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.436609 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41e00-d519-45b2-868c-edfff1b0449f-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:21.939379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:21.939343 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:22.069900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.069874 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" Oct 30 09:30:22.070279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.069873 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc" event={"ID":"6dc41e00-d519-45b2-868c-edfff1b0449f","Type":"ContainerDied","Data":"23a1d2c120670785861c3a747d0603d41429c9db423c06b34bb3ad16d461b651"} Oct 30 09:30:22.070279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.070011 4920 scope.go:122] "RemoveContainer" containerID="0875edf7d720c98e0f937cfb6f2af1ac24a109ddef6a3e392587cc490c8d93c9" Oct 30 09:30:22.151475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.151381 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:22.156547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.156424 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-15e1cc-29889210-lfxgc"] Oct 30 09:30:22.256534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.256499 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:22.353576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.353542 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\") pod \"ca164d4e-9296-4e9a-8b86-40e773270ce6\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " Oct 30 09:30:22.353775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.353762 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\" (UniqueName: \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\") pod \"ca164d4e-9296-4e9a-8b86-40e773270ce6\" (UID: \"ca164d4e-9296-4e9a-8b86-40e773270ce6\") " Oct 30 09:30:22.353993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.353967 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca" pod "ca164d4e-9296-4e9a-8b86-40e773270ce6" (UID: "ca164d4e-9296-4e9a-8b86-40e773270ce6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:22.354200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.354186 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca164d4e-9296-4e9a-8b86-40e773270ce6-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:22.355878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.355852 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6" pod "ca164d4e-9296-4e9a-8b86-40e773270ce6" (UID: "ca164d4e-9296-4e9a-8b86-40e773270ce6"). InnerVolumeSpecName "kube-api-access-gcft6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:22.399618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.399581 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" path="/var/lib/kubelet/pods/6dc41e00-d519-45b2-868c-edfff1b0449f/volumes" Oct 30 09:30:22.456258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.456207 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcft6\" (UniqueName: \"kubernetes.io/projected/ca164d4e-9296-4e9a-8b86-40e773270ce6-kube-api-access-gcft6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:22.808379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:22.808346 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:30:23.079012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.078925 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" Oct 30 09:30:23.079012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.078964 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r" event={"ID":"ca164d4e-9296-4e9a-8b86-40e773270ce6","Type":"ContainerDied","Data":"688dd5ab9fd5d0c3fce90f96c1e9d30657277873df859b34312dae0862413e32"} Oct 30 09:30:23.079547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.079020 4920 scope.go:122] "RemoveContainer" containerID="0df094e118389a11057c5da81c0f674049640eac95fc83fa04a0b8e2715ab7aa" Oct 30 09:30:23.151119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.151083 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:23.173280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.173245 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889210-6r78r"] Oct 30 09:30:23.673240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:23.659085 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:24.474240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:24.470203 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" path="/var/lib/kubelet/pods/ca164d4e-9296-4e9a-8b86-40e773270ce6/volumes" Oct 30 09:30:25.274240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:25.273320 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:30:25.274240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:25.273564 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" podUID="ec2e7195-10fb-44e1-8333-c93b8a80e584" containerName="maas-api" containerID="cri-o://2da331e3367cb24771b59476e749af79df8216c2a9f5b0f90ffd51b4d9112093" gracePeriod=30 Oct 30 09:30:25.397020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:25.396972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:26.128130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.128091 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:26.204052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.203960 4920 generic.go:356] "Generic (PLEG): container finished" podID="ec2e7195-10fb-44e1-8333-c93b8a80e584" containerID="2da331e3367cb24771b59476e749af79df8216c2a9f5b0f90ffd51b4d9112093" exitCode=0 Oct 30 09:30:26.204052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.204034 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" event={"ID":"ec2e7195-10fb-44e1-8333-c93b8a80e584","Type":"ContainerDied","Data":"2da331e3367cb24771b59476e749af79df8216c2a9f5b0f90ffd51b4d9112093"} Oct 30 09:30:26.251555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.251529 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:30:26.330405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.330369 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\" (UniqueName: \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\") pod \"ec2e7195-10fb-44e1-8333-c93b8a80e584\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " Oct 30 09:30:26.330606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.330474 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\") pod \"ec2e7195-10fb-44e1-8333-c93b8a80e584\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " Oct 30 09:30:26.330713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.330694 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\") pod \"ec2e7195-10fb-44e1-8333-c93b8a80e584\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " Oct 30 09:30:26.330784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.330768 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") pod \"ec2e7195-10fb-44e1-8333-c93b8a80e584\" (UID: \"ec2e7195-10fb-44e1-8333-c93b8a80e584\") " Oct 30 09:30:26.331161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.331135 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca" pod "ec2e7195-10fb-44e1-8333-c93b8a80e584" (UID: "ec2e7195-10fb-44e1-8333-c93b8a80e584"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:26.331398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.331378 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2e7195-10fb-44e1-8333-c93b8a80e584-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:26.333913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.333853 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls" pod "ec2e7195-10fb-44e1-8333-c93b8a80e584" (UID: "ec2e7195-10fb-44e1-8333-c93b8a80e584"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:26.333913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.333875 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs" pod "ec2e7195-10fb-44e1-8333-c93b8a80e584" (UID: "ec2e7195-10fb-44e1-8333-c93b8a80e584"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:26.335178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.335152 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl" pod "ec2e7195-10fb-44e1-8333-c93b8a80e584" (UID: "ec2e7195-10fb-44e1-8333-c93b8a80e584"). InnerVolumeSpecName "kube-api-access-qt4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:26.349892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.349867 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:26.438157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.438120 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:26.438396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.438382 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2e7195-10fb-44e1-8333-c93b8a80e584-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:26.438697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.438680 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt4rl\" (UniqueName: \"kubernetes.io/projected/ec2e7195-10fb-44e1-8333-c93b8a80e584-kube-api-access-qt4rl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:26.444350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.444311 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt"] Oct 30 09:30:26.450774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.450741 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c9232f-b7pnt"] Oct 30 09:30:26.536616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.536582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:26.706689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.706637 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:30:26.707483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707457 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ec2e7195-10fb-44e1-8333-c93b8a80e584" containerName="maas-api" Oct 30 09:30:26.707483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707481 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2e7195-10fb-44e1-8333-c93b8a80e584" containerName="maas-api" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707497 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707502 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707508 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707514 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707528 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707535 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707552 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707557 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707568 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707573 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707590 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707595 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707602 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707608 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707628 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f8f22005-cec9-45c0-a3a8-674d16e0bce3" containerName="revoke-keys" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707633 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f22005-cec9-45c0-a3a8-674d16e0bce3" containerName="revoke-keys" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707652 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707657 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707665 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.707706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707669 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707758 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707769 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707781 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e969a6e-ab0f-49c0-bd19-9c8670209bdc" containerName="istio-proxy" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707790 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707797 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707803 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd4f29d-c23f-434b-a52d-c30df3f9618e" containerName="payload-pre-processing" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707809 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707814 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f22005-cec9-45c0-a3a8-674d16e0bce3" containerName="revoke-keys" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707819 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707826 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0538287e-480f-4c4b-98f8-ce86b0b0fa43" containerName="payload-processing" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707832 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2e7195-10fb-44e1-8333-c93b8a80e584" containerName="maas-api" Oct 30 09:30:26.708761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.707837 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.712170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.712143 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.739362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.739324 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:30:26.740532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.740471 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.740725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.740709 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.740856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.740845 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.740931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.740920 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41e00-d519-45b2-868c-edfff1b0449f" containerName="cleanup" Oct 30 09:30:26.741034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.741025 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.741170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.741158 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.741444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.741430 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f883161b-2344-4426-9fd0-b479d0c98bc2" containerName="cleanup" Oct 30 09:30:26.741546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.741536 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca164d4e-9296-4e9a-8b86-40e773270ce6" containerName="cleanup" Oct 30 09:30:26.745943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.745922 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:30:26.746143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.746054 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.750610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.750584 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:30:26.755059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.755018 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3a61e1-openshift-default-dockercfg-vvdt5\"" Oct 30 09:30:26.755380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.755360 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.762758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.761161 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:30:26.767145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.767113 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:30:26.846290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846177 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.846481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846300 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.846481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846340 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.846481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846435 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.846656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846501 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.846656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846561 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.846656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846606 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmq2\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.846656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846633 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.846941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846708 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.846941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846739 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.846941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846785 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464p9\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.846941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846870 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.846941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846912 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.847263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.846946 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847035 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847074 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.847263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847136 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847259 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847310 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847364 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847407 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847451 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847488 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.847527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847518 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.847905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847549 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.847905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847579 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.847905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.847604 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvb4\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949129 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949199 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949264 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949287 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949308 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949330 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949349 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvb4\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949549 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949592 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.949718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949639 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.949718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949694 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949730 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949784 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949759 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949761 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.949877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949797 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.949877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949849 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmq2\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949879 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949926 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.949971 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950021 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464p9\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950058 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950104 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950135 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950171 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950164 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950266 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950308 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950366 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950396 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950424 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950505 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950526 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950741 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.950903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950794 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.950903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950801 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.950903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.950830 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.951190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951094 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.951305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951283 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.951372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951333 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.951430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951416 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.951545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951500 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.951545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.951526 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.952767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.952397 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.952767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.952723 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.953474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.953453 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.954016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.953995 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.954404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.954383 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.954706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.954687 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.969277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.969164 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.969705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.969677 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.969841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.969821 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvb4\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:26.970271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.970252 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464p9\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:26.970371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.970345 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmq2\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:26.970996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:26.970973 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:27.043895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.043860 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:27.075855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.075821 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:27.088318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.086844 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:27.094602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.094567 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:27.227327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.226516 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" Oct 30 09:30:27.227327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.226514 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t" event={"ID":"ec2e7195-10fb-44e1-8333-c93b8a80e584","Type":"ContainerDied","Data":"9ab261f6bf96234ed0c6be38c119a68898b02b78db79266b34244c78fb381b7f"} Oct 30 09:30:27.227327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.226669 4920 scope.go:122] "RemoveContainer" containerID="2da331e3367cb24771b59476e749af79df8216c2a9f5b0f90ffd51b4d9112093" Oct 30 09:30:27.282416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.282339 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:30:27.286490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.286462 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c9232f-6c4c6dbbdb-mzl2t"] Oct 30 09:30:27.374462 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:27.374425 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cfc529_10a0_4e9d_b7ff_7c23b346da87.slice/crio-7ac05789bb4a5118899e0da740db3ffc854481293666890794a6e6fb3b57b856 WatchSource:0}: Error finding container 7ac05789bb4a5118899e0da740db3ffc854481293666890794a6e6fb3b57b856: Status 404 returned error can't find the container with id 7ac05789bb4a5118899e0da740db3ffc854481293666890794a6e6fb3b57b856 Oct 30 09:30:27.376763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.376738 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:30:27.377048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.377019 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.377089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.377080 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.377136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.377111 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.418157 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:27.418113 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85f20d6_08cb_4112_8200_c84d9abf97f5.slice/crio-b8e77b25e13f3d2de73a615a0a74206d789e5be87fe237b79bd2120567941e36 WatchSource:0}: Error finding container b8e77b25e13f3d2de73a615a0a74206d789e5be87fe237b79bd2120567941e36: Status 404 returned error can't find the container with id b8e77b25e13f3d2de73a615a0a74206d789e5be87fe237b79bd2120567941e36 Oct 30 09:30:27.419291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.419263 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:30:27.421409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.421375 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.421503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.421439 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.421503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.421468 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.447638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.447612 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:30:27.451368 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:27.451335 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2e19e1_79b1_4bb2_9bab_682aaa7599aa.slice/crio-a609a9379a934bc4f200a772a47fea3d7911b80f7a534a0ab1b9fbd46e3ec330 WatchSource:0}: Error finding container a609a9379a934bc4f200a772a47fea3d7911b80f7a534a0ab1b9fbd46e3ec330: Status 404 returned error can't find the container with id a609a9379a934bc4f200a772a47fea3d7911b80f7a534a0ab1b9fbd46e3ec330 Oct 30 09:30:27.453660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.453630 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.453753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.453692 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.453753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:27.453718 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:28.279128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.271088 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" event={"ID":"28cfc529-10a0-4e9d-b7ff-7c23b346da87","Type":"ContainerStarted","Data":"c4b71cc9bb0bb01b7d9dfb1d26fb68dfff2e09e726a5a1f924fff4551ec277a9"} Oct 30 09:30:28.279128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.271199 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" event={"ID":"28cfc529-10a0-4e9d-b7ff-7c23b346da87","Type":"ContainerStarted","Data":"7ac05789bb4a5118899e0da740db3ffc854481293666890794a6e6fb3b57b856"} Oct 30 09:30:28.295455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.295410 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" event={"ID":"f85f20d6-08cb-4112-8200-c84d9abf97f5","Type":"ContainerStarted","Data":"0f7085b948442d6fc089057e2c96cf08b36d514df56fe0b232986ff47b713872"} Oct 30 09:30:28.295636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.295479 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" event={"ID":"f85f20d6-08cb-4112-8200-c84d9abf97f5","Type":"ContainerStarted","Data":"b8e77b25e13f3d2de73a615a0a74206d789e5be87fe237b79bd2120567941e36"} Oct 30 09:30:28.319242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.303337 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" event={"ID":"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa","Type":"ContainerStarted","Data":"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2"} Oct 30 09:30:28.319242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.303403 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" event={"ID":"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa","Type":"ContainerStarted","Data":"a609a9379a934bc4f200a772a47fea3d7911b80f7a534a0ab1b9fbd46e3ec330"} Oct 30 09:30:28.378237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.374156 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podStartSLOduration=2.374132369 podStartE2EDuration="2.374132369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:28.322555687 +0000 UTC m=+2192.045982832" watchObservedRunningTime="2026-10-30 09:30:28.374132369 +0000 UTC m=+2192.097559516" Oct 30 09:30:28.386788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.386723 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podStartSLOduration=2.386703979 podStartE2EDuration="2.386703979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:28.382757618 +0000 UTC m=+2192.106184778" watchObservedRunningTime="2026-10-30 09:30:28.386703979 +0000 UTC m=+2192.110131125" Oct 30 09:30:28.469107 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.469058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:28.497257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.489564 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fdfa40-f7d2-4ad3-b079-ae663af267f9" path="/var/lib/kubelet/pods/45fdfa40-f7d2-4ad3-b079-ae663af267f9/volumes" Oct 30 09:30:28.497257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:28.490287 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2e7195-10fb-44e1-8333-c93b8a80e584" path="/var/lib/kubelet/pods/ec2e7195-10fb-44e1-8333-c93b8a80e584/volumes" Oct 30 09:30:29.048243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.046547 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:29.069241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.062355 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:29.069241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.062427 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:29.091243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.088361 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:29.091243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.089095 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:29.091243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.089147 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:29.091243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.089172 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:29.092631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.092440 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:29.092631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:29.092498 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:30.057774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.057737 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:30.058548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.058512 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:30.085016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.084980 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:30.085801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.085761 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:30.088667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.088573 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:30.088667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.088615 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:30.626242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:30.619174 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:31.054241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.044706 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:31.054241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.044768 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:31.077237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.076733 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:31.077237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.076799 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:31.088915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.088880 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:31.089203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:31.089176 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:31.306296 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:31.304619 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:30:32.092250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.089829 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podStartSLOduration=6.089808617 podStartE2EDuration="6.089808617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:28.45884717 +0000 UTC m=+2192.182274314" watchObservedRunningTime="2026-10-30 09:30:32.089808617 +0000 UTC m=+2195.813235760" Oct 30 09:30:32.096377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.095346 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:32.096377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.095414 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:32.097203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.096713 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:32.097203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.096762 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:32.097203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.096844 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:32.097203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.096903 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:32.270476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.270430 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:30:32.400297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.363389 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:32.606942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:32.606887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:33.045129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.045087 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:33.045684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.045651 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:33.090155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.085007 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:33.090155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.085082 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:33.090155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.089664 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:33.090155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:33.089716 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:34.046239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.045886 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:30:34.046239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.045949 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 09:30:34.076389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.076344 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 09:30:34.076615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.076418 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 09:30:34.087640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.087583 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 09:30:34.087833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.087645 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 09:30:34.243609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:34.243505 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:35.609819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.609790 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:35.611067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.611041 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:35.613065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.613045 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:30:35.629920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.628096 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:35.629920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.629502 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:35.631649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.631629 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:30:35.871617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.871527 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:35.880275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.879126 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:35.881837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.881808 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:30:35.926934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:35.926899 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:37.613768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:37.613729 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:38.572408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:38.572328 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:39.173168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:39.173130 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:39.757644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:39.757597 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:40.237977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:40.237925 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:40.523970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:40.523883 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:40.842323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:40.842288 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:41.720365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.720320 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:30:41.726059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.726032 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.745680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.745639 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:30:41.813455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813411 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813473 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813565 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813594 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813693 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813764 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813820 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.813881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813858 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.814113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.813896 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915442 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915498 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915528 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915553 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915616 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915699 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915773 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915802 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.915923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.915848 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.916133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.916064 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.916133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.916104 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.916468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.916446 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.916594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.916571 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.916930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.916901 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.918065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.918043 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.918571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.918551 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.934605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.934576 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:41.934817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:41.934793 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:42.051306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.049492 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:42.403596 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:42.403563 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107f7c37_3e30_4b2d_919b_9b0a630e3e60.slice/crio-14a810a0c9964ad43f64bb785a64bd5607bbae9e8b23f59acf4db10f06787ca3 WatchSource:0}: Error finding container 14a810a0c9964ad43f64bb785a64bd5607bbae9e8b23f59acf4db10f06787ca3: Status 404 returned error can't find the container with id 14a810a0c9964ad43f64bb785a64bd5607bbae9e8b23f59acf4db10f06787ca3 Oct 30 09:30:42.406389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.406367 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:30:42.406776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.406743 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:42.406862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.406819 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:42.406922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.406865 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:42.435430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.435394 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:42.977151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.977110 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" event={"ID":"107f7c37-3e30-4b2d-919b-9b0a630e3e60","Type":"ContainerStarted","Data":"7874f04d6adab9664872587cf430c4b6545554d704259b5c8fc1c29a83ab051e"} Oct 30 09:30:42.977610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:42.977178 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" event={"ID":"107f7c37-3e30-4b2d-919b-9b0a630e3e60","Type":"ContainerStarted","Data":"14a810a0c9964ad43f64bb785a64bd5607bbae9e8b23f59acf4db10f06787ca3"} Oct 30 09:30:43.050866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:43.050833 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:43.052640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:43.052613 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:43.052758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:43.052660 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:44.050966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.050933 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:44.051486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.050990 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:44.232036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.231990 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:44.336598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.336416 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:44.440078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.440039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:44.541683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:44.541642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:45.050363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:45.050324 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:45.050595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:45.050402 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:45.891880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:45.891827 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:46.054239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:46.053400 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:46.054239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:46.053459 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:46.107241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:46.097476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:46.230297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:46.230185 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:47.050654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:47.050608 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:47.051137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:47.051110 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:47.985398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:47.985257 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerName="main" containerID="cri-o://a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86" gracePeriod=45 Oct 30 09:30:48.050812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.050313 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:48.050812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.050418 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:48.080477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.080441 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused" Oct 30 09:30:48.089745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.089705 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:48.446530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.446496 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:30:48.489394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.489343 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podStartSLOduration=7.489327129 podStartE2EDuration="7.489327129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:43.026447531 +0000 UTC m=+2206.749874695" watchObservedRunningTime="2026-10-30 09:30:48.489327129 +0000 UTC m=+2212.212754271" Oct 30 09:30:48.639154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639090 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.639421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639207 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\" (UniqueName: \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.639421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639362 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.639557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639424 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.639557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639473 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.639670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639583 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\") pod \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\" (UID: \"4dfa1636-fa00-4892-82a9-2cbfdb3b8680\") " Oct 30 09:30:48.640422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.639996 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:48.640422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.640021 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:48.640422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.640127 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:48.640422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.640407 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:48.640665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.640430 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:48.640665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.640444 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:48.643136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.642857 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "kube-api-access-lwjrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:48.643332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.643309 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:48.643600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.643573 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm" pod "4dfa1636-fa00-4892-82a9-2cbfdb3b8680" (UID: "4dfa1636-fa00-4892-82a9-2cbfdb3b8680"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:48.742188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.742143 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:48.742188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.742189 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:48.742436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:48.742203 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjrw\" (UniqueName: \"kubernetes.io/projected/4dfa1636-fa00-4892-82a9-2cbfdb3b8680-kube-api-access-lwjrw\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:30:49.050516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.050482 4920 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 09:30:49.050714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.050539 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 09:30:49.086722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.086688 4920 generic.go:356] "Generic (PLEG): container finished" podID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerID="a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86" exitCode=0 Oct 30 09:30:49.087175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.086753 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" Oct 30 09:30:49.087175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.086761 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" event={"ID":"4dfa1636-fa00-4892-82a9-2cbfdb3b8680","Type":"ContainerDied","Data":"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86"} Oct 30 09:30:49.087175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.086804 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf" event={"ID":"4dfa1636-fa00-4892-82a9-2cbfdb3b8680","Type":"ContainerDied","Data":"59446b2b5de5fe48e7cfba7baa8eb8b83d6742798cbe0396238b9c9933024ecb"} Oct 30 09:30:49.087175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.086824 4920 scope.go:122] "RemoveContainer" containerID="a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86" Oct 30 09:30:49.101211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.101178 4920 scope.go:122] "RemoveContainer" containerID="a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86" Oct 30 09:30:49.101641 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:30:49.101614 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86\": container with ID starting with a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86 not found: ID does not exist" containerID="a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86" Oct 30 09:30:49.101813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.101779 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86"} err="failed to get container status \"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86\": rpc error: code = NotFound desc = could not find container \"a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86\": container with ID starting with a1b9b2c79cb19c4fa1d31ea209756d442635c39793eb06372b26dcd5295f8f86 not found: ID does not exist" Oct 30 09:30:49.141583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.141544 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:30:49.151192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.151164 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c9232f/praxis-route-c9232f-kserve-8db8bf5c9-z48tf"] Oct 30 09:30:49.692802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:49.692760 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:50.054851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:50.054818 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:50.055277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:50.055251 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:50.056159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:50.056142 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:30:50.399171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:50.399082 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" path="/var/lib/kubelet/pods/4dfa1636-fa00-4892-82a9-2cbfdb3b8680/volumes" Oct 30 09:30:50.577212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:50.577183 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:51.810610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:51.810561 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:54.452356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:54.452301 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.596398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:54.596327 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.628095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:54.628045 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.710472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:54.710365 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.945514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:54.944329 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:55.036601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:55.036551 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:55.092982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:55.092919 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:56.995527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:56.995490 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:57.079488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.079442 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw"] Oct 30 09:30:57.080559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.080529 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerName="main" Oct 30 09:30:57.080559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.080559 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerName="main" Oct 30 09:30:57.080848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.080825 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa1636-fa00-4892-82a9-2cbfdb3b8680" containerName="main" Oct 30 09:30:57.085595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.085569 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.098294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.098257 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw"] Oct 30 09:30:57.114166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.112459 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:57.114166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.112643 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:30:57.229574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.229534 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjscr\" (UniqueName: \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.229761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.229590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.229761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.229685 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.331483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.331443 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjscr\" (UniqueName: \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.331483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.331486 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.331729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.331516 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.332140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.332117 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.334056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.334027 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.350435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.350404 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjscr\" (UniqueName: \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\") pod \"maas-api-revoke-sub-677fce03-fp2rw\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.422505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.422472 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:30:57.737025 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:30:57.736412 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec82776_9277_4509_84a5_2138e307f5a7.slice/crio-361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0 WatchSource:0}: Error finding container 361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0: Status 404 returned error can't find the container with id 361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0 Oct 30 09:30:57.737282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:57.737181 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw"] Oct 30 09:30:58.261841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:58.261744 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" event={"ID":"9ec82776-9277-4509-84a5-2138e307f5a7","Type":"ContainerStarted","Data":"6782828345eb65dd7f438d813e83c462fb8f7e42d435a6d3e2406d2908bc8454"} Oct 30 09:30:58.261841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:58.261809 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" event={"ID":"9ec82776-9277-4509-84a5-2138e307f5a7","Type":"ContainerStarted","Data":"361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0"} Oct 30 09:30:58.307549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:58.307486 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" podStartSLOduration=1.3074724340000001 podStartE2EDuration="1.307472434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:58.303120699 +0000 UTC m=+2222.026547845" watchObservedRunningTime="2026-10-30 09:30:58.307472434 +0000 UTC m=+2222.030899577" Oct 30 09:30:58.700417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:58.700373 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:30:59.270665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:59.270624 4920 generic.go:356] "Generic (PLEG): container finished" podID="9ec82776-9277-4509-84a5-2138e307f5a7" containerID="6782828345eb65dd7f438d813e83c462fb8f7e42d435a6d3e2406d2908bc8454" exitCode=0 Oct 30 09:30:59.271046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:30:59.270717 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" event={"ID":"9ec82776-9277-4509-84a5-2138e307f5a7","Type":"ContainerDied","Data":"6782828345eb65dd7f438d813e83c462fb8f7e42d435a6d3e2406d2908bc8454"} Oct 30 09:31:00.433069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.433032 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:00.475628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.475594 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:31:00.576239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.576188 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\") pod \"9ec82776-9277-4509-84a5-2138e307f5a7\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " Oct 30 09:31:00.576429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.576310 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\" (UniqueName: \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\") pod \"9ec82776-9277-4509-84a5-2138e307f5a7\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " Oct 30 09:31:00.576429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.576361 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\") pod \"9ec82776-9277-4509-84a5-2138e307f5a7\" (UID: \"9ec82776-9277-4509-84a5-2138e307f5a7\") " Oct 30 09:31:00.576668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.576637 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca" pod "9ec82776-9277-4509-84a5-2138e307f5a7" (UID: "9ec82776-9277-4509-84a5-2138e307f5a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:00.576973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.576951 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:00.578793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.578762 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca" pod "9ec82776-9277-4509-84a5-2138e307f5a7" (UID: "9ec82776-9277-4509-84a5-2138e307f5a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:00.582896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.582867 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr" pod "9ec82776-9277-4509-84a5-2138e307f5a7" (UID: "9ec82776-9277-4509-84a5-2138e307f5a7"). InnerVolumeSpecName "kube-api-access-fjscr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:00.677846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.677803 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjscr\" (UniqueName: \"kubernetes.io/projected/9ec82776-9277-4509-84a5-2138e307f5a7-kube-api-access-fjscr\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:00.677846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:00.677838 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec82776-9277-4509-84a5-2138e307f5a7-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:01.135500 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:31:01.135458 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:31:01.288540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:01.288502 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" Oct 30 09:31:01.288828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:01.288542 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw" event={"ID":"9ec82776-9277-4509-84a5-2138e307f5a7","Type":"ContainerDied","Data":"361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0"} Oct 30 09:31:01.288828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:01.288589 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361cf6f3528f76a1dfbb0626d2bb9b7eaf96c80946bd78e9d356eaea20cad5a0" Oct 30 09:31:02.084280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:02.084241 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:02.192485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:02.192434 4920 scope.go:122] "RemoveContainer" containerID="02f172203dd5cfb2ab09b327f802d191d5511ecff0743c093e40c211e484a960" Oct 30 09:31:02.212250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:02.212180 4920 scope.go:122] "RemoveContainer" containerID="95e6c37b30cc6d7a5788c379ff0c2896c553c46d300e302297b806ae90092134" Oct 30 09:31:03.672447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.672407 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:31:03.673438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.673408 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9ec82776-9277-4509-84a5-2138e307f5a7" containerName="revoke-keys" Oct 30 09:31:03.673438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.673437 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec82776-9277-4509-84a5-2138e307f5a7" containerName="revoke-keys" Oct 30 09:31:03.673644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.673614 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec82776-9277-4509-84a5-2138e307f5a7" containerName="revoke-keys" Oct 30 09:31:03.679173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.679152 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.697327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.697293 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-066f95-serving-cert\"" Oct 30 09:31:03.702696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.702649 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:31:03.750492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.750460 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:03.825733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.825691 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.825936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.825763 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.825936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.825912 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnq2\" (UniqueName: \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.826059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.826043 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.850748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.850713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:03.933317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.933209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnq2\" (UniqueName: \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.933998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.933972 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.934312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.934293 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.934478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.934462 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.935271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.935245 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.941189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.941158 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.954873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.954838 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.957716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.957083 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnq2\" (UniqueName: \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\") pod \"maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:03.958667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.958593 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:03.998071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:03.998033 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:04.304334 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:31:04.304294 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaea09ae_2903_4dbc_bbc7_2583ad7f2508.slice/crio-4e24d63ed215447dd6261671a9075c0b3a7775522337a686cad151e25ac49418 WatchSource:0}: Error finding container 4e24d63ed215447dd6261671a9075c0b3a7775522337a686cad151e25ac49418: Status 404 returned error can't find the container with id 4e24d63ed215447dd6261671a9075c0b3a7775522337a686cad151e25ac49418 Oct 30 09:31:04.306053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:04.306035 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:31:04.307172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:04.307147 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:31:04.316656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:04.316623 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" event={"ID":"eaea09ae-2903-4dbc-bbc7-2583ad7f2508","Type":"ContainerStarted","Data":"4e24d63ed215447dd6261671a9075c0b3a7775522337a686cad151e25ac49418"} Oct 30 09:31:05.326150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:05.326106 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" event={"ID":"eaea09ae-2903-4dbc-bbc7-2583ad7f2508","Type":"ContainerStarted","Data":"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236"} Oct 30 09:31:05.326596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:05.326204 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:05.375509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:05.375443 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" podStartSLOduration=1.964474608 podStartE2EDuration="2.37542324s" totalImagesPullingTime="410.948632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:05.369712808 +0000 UTC m=+2229.093139950" watchObservedRunningTime="2026-10-30 09:31:05.37542324 +0000 UTC m=+2229.098850384" Oct 30 09:31:05.575007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:05.574971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:07.366277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:07.366238 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:09.172957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:09.172920 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:09.695098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:09.695059 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:10.230232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:10.230178 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:11.298575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:11.298539 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:11.340937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:11.340908 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:31:13.379629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:13.379588 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:14.946447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:14.946399 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:15.276293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:15.275363 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:31:16.623592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:16.623534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:16.740731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:16.740259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:17.074298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:17.044171 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:17.147242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:17.116028 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:31:17.299635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:17.147780 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" containerID="cri-o://1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac" gracePeriod=30 Oct 30 09:31:17.384535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:17.290969 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:17.911374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:17.911318 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.041385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.041336 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.042400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.041394 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.047397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.047353 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:31:18.438360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.438310 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.464441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.464375 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.580561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.579341 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.614502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.614453 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:18.640351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:18.640302 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:19.049091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:19.049044 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:19.050238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:19.050196 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:31:19.371327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:19.371060 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:19.461964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:19.461932 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:31:20.281840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:20.281800 4920 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:31:20.282286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:20.281868 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:31:21.052933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:21.052896 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:22.656895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.656868 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:31:22.712418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712320 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712404 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712459 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712525 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712615 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712648 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:22.712953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712687 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712781 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712841 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.712953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.712892 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\") pod \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\" (UID: \"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b\") " Oct 30 09:31:22.713551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.713507 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.713551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.713532 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:22.713750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.713615 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:22.713750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.713700 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:22.713750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.713732 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:22.716270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.716208 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:31:22.716492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.716464 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:22.716615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.716518 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "kube-api-access-dhtmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:22.719090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.719061 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy" pod "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" (UID: "c4445a69-4edc-4ffb-b230-0d97bfc7ea9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:22.776014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.775980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:22.815445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815412 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815439 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815448 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815457 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtmx\" (UniqueName: \"kubernetes.io/projected/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-kube-api-access-dhtmx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815467 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815475 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815483 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.815759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.815492 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:22.822624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.822593 4920 generic.go:356] "Generic (PLEG): container finished" podID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerID="1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac" exitCode=0 Oct 30 09:31:22.822819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.822649 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" Oct 30 09:31:22.822819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.822680 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerDied","Data":"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac"} Oct 30 09:31:22.822819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.822741 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk" event={"ID":"c4445a69-4edc-4ffb-b230-0d97bfc7ea9b","Type":"ContainerDied","Data":"077098f57850116e54be2841597dd56f79f86a1e1ce354892367ef1045466005"} Oct 30 09:31:22.822819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.822772 4920 scope.go:122] "RemoveContainer" containerID="1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac" Oct 30 09:31:22.882417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.882389 4920 scope.go:122] "RemoveContainer" containerID="f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc" Oct 30 09:31:22.893304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.893268 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:31:22.895770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.895751 4920 scope.go:122] "RemoveContainer" containerID="1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac" Oct 30 09:31:22.896119 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:31:22.896093 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac\": container with ID starting with 1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac not found: ID does not exist" containerID="1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac" Oct 30 09:31:22.896280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.896125 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac"} err="failed to get container status \"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac\": rpc error: code = NotFound desc = could not find container \"1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac\": container with ID starting with 1d068f770a586f8ecf63d1ddd4af7fb659903f33e003e2ed5c129d288fc51eac not found: ID does not exist" Oct 30 09:31:22.896280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.896145 4920 scope.go:122] "RemoveContainer" containerID="f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc" Oct 30 09:31:22.896526 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:31:22.896460 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc\": container with ID starting with f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc not found: ID does not exist" containerID="f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc" Oct 30 09:31:22.896638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.896531 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc"} err="failed to get container status \"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc\": rpc error: code = NotFound desc = could not find container \"f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc\": container with ID starting with f862c325c0b900c4e531bceda3cd80a8e8dcfae0bd74b69bd7501cddc69038fc not found: ID does not exist" Oct 30 09:31:22.897666 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:22.897646 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-q4zkk"] Oct 30 09:31:23.798793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:23.798749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:23.899619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:23.899585 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:24.399600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:24.399563 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" path="/var/lib/kubelet/pods/c4445a69-4edc-4ffb-b230-0d97bfc7ea9b/volumes" Oct 30 09:31:24.659629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:24.659528 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:26.237265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:26.237227 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:28.137790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:28.137753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:30.136602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:30.136560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:30.579682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:30.579642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:31.136844 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:31:31.136803 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:31:32.081654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.081610 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9"] Oct 30 09:31:32.082351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.082329 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:32.082351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.082351 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:32.082529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.082436 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:32.082529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.082445 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:32.082636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.082595 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:32.087822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.087799 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.105422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.105398 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:32.108937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.108907 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:32.121761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.121727 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9"] Oct 30 09:31:32.162893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.162860 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.163697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.163664 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.163814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.163722 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh8v\" (UniqueName: \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.265192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.265152 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.265389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.265201 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh8v\" (UniqueName: \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.265389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.265313 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.266030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.266002 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.267637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.267616 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.286478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.286449 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh8v\" (UniqueName: \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\") pod \"maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.402454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.402363 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:32.631394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.631358 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:32.724310 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:31:32.720927 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b78f4c_9e32_4b5a_a67e_3fbb9dc67e86.slice/crio-d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3 WatchSource:0}: Error finding container d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3: Status 404 returned error can't find the container with id d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3 Oct 30 09:31:32.724310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.723042 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9"] Oct 30 09:31:32.761489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.759842 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:32.911709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:32.911669 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" event={"ID":"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86","Type":"ContainerStarted","Data":"d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3"} Oct 30 09:31:33.922041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:33.922007 4920 generic.go:356] "Generic (PLEG): container finished" podID="36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" containerID="8f665683f76e8db9ee49d673994e4f08bda200e258c481e83ab2b912496de13c" exitCode=0 Oct 30 09:31:33.922502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:33.922091 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" event={"ID":"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86","Type":"ContainerDied","Data":"8f665683f76e8db9ee49d673994e4f08bda200e258c481e83ab2b912496de13c"} Oct 30 09:31:35.057681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.057643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:35.155242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.148671 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:35.236587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.236483 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\") pod \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " Oct 30 09:31:35.236781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.236595 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\") pod \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " Oct 30 09:31:35.236884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.236866 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\" (UniqueName: \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\") pod \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\" (UID: \"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86\") " Oct 30 09:31:35.237332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.237300 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca" pod "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" (UID: "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:35.237521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.237498 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:35.239978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.239952 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca" pod "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" (UID: "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:35.240133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.239987 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v" pod "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" (UID: "36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86"). InnerVolumeSpecName "kube-api-access-chh8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:35.339002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.338957 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chh8v\" (UniqueName: \"kubernetes.io/projected/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-kube-api-access-chh8v\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:35.339002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.339000 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:35.945438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.945405 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" event={"ID":"36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86","Type":"ContainerDied","Data":"d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3"} Oct 30 09:31:35.945438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.945430 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9" Oct 30 09:31:35.945438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:35.945454 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e323a5178428131026c137b28146f0aaa15791a417200fa32eea6bb5b807a3" Oct 30 09:31:36.786783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:36.786749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:38.520132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:38.520097 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:40.211733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:40.211697 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:40.256051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:40.255192 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:31:42.463243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:42.456398 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.472241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:42.469301 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.580461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:42.580416 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.595353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:42.595294 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.949240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:42.948326 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.435294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.427637 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:43.493264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.465313 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:31:43.493264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.465908 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:31:43.512721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.512336 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 09:31:43.512721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.512650 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:31:43.513060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.513036 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:31:43.628238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.620750 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.628238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.621038 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:31:43.654241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.629792 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.667383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.630088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:43.929121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.929077 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 09:31:43.951959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.951908 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.952967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:43.952942 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:31:44.471235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:44.465670 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:44.490122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:44.490088 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:31:44.490334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:44.490172 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:31:44.490334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:44.490205 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:31:44.490469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:44.490407 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:31:45.011835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:45.011794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:45.683037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:45.683007 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:45.804473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:45.804435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:45.908114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:45.908075 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:46.326067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:46.326031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:47.718535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:47.718499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:48.748622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:48.748593 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6_c1c46a36-4a8d-4db2-bc02-be1e87f52b13/payload-processing/0.log" Oct 30 09:31:49.447805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:49.442861 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:50.274212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.274177 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44"] Oct 30 09:31:50.274865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.274847 4920 cpu_manager.go:414] "containerMap: removing container" podUID="36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" containerName="revoke-keys" Oct 30 09:31:50.274924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.274865 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" containerName="revoke-keys" Oct 30 09:31:50.274967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.274960 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4445a69-4edc-4ffb-b230-0d97bfc7ea9b" containerName="istio-proxy" Oct 30 09:31:50.275003 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.274973 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" containerName="revoke-keys" Oct 30 09:31:50.279725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.279703 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.298976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.298948 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44"] Oct 30 09:31:50.305535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.305507 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:31:50.305684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.305640 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:31:50.413540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.413489 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.413731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.413567 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.413731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.413624 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwcf\" (UniqueName: \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.515040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.515000 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.515264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.515059 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.515264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.515115 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwcf\" (UniqueName: \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.515719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.515696 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.517367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.517345 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.534578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.534554 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwcf\" (UniqueName: \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.597707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.597669 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:50.697500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.697445 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:50.908814 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:31:50.908780 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78494d89_3baa_4009_a153_aefff1ba4597.slice/crio-7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd WatchSource:0}: Error finding container 7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd: Status 404 returned error can't find the container with id 7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd Oct 30 09:31:50.911232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:50.911188 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44"] Oct 30 09:31:51.489157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:51.489067 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:51.608073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:51.608035 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" event={"ID":"78494d89-3baa-4009-a153-aefff1ba4597","Type":"ContainerStarted","Data":"53d99a197a52af98f0d57ea557cf027f3b921e2fed787ea241d00c30368101d0"} Oct 30 09:31:51.608408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:51.608384 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" event={"ID":"78494d89-3baa-4009-a153-aefff1ba4597","Type":"ContainerStarted","Data":"7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd"} Oct 30 09:31:51.647957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:51.647882 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" podStartSLOduration=1.6478591599999999 podStartE2EDuration="1.64785916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:51.645198445 +0000 UTC m=+2275.368625588" watchObservedRunningTime="2026-10-30 09:31:51.64785916 +0000 UTC m=+2275.371286305" Oct 30 09:31:52.094321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.094289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:52.204780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.200873 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:52.436389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.436351 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:31:52.444692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.444647 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" containerID="cri-o://58d366b7d38f6081ac205cf8fbc3be66faa0def689e13aad6c49f95651e5df5e" gracePeriod=30 Oct 30 09:31:52.447095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.447062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:52.685406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.682538 4920 generic.go:356] "Generic (PLEG): container finished" podID="78494d89-3baa-4009-a153-aefff1ba4597" containerID="53d99a197a52af98f0d57ea557cf027f3b921e2fed787ea241d00c30368101d0" exitCode=0 Oct 30 09:31:52.685406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:52.682789 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" event={"ID":"78494d89-3baa-4009-a153-aefff1ba4597","Type":"ContainerDied","Data":"53d99a197a52af98f0d57ea557cf027f3b921e2fed787ea241d00c30368101d0"} Oct 30 09:31:53.344980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.344938 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:53.881495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.881464 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:53.995673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.995586 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\") pod \"78494d89-3baa-4009-a153-aefff1ba4597\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " Oct 30 09:31:53.995838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.995824 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\") pod \"78494d89-3baa-4009-a153-aefff1ba4597\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " Oct 30 09:31:53.995882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.995858 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\" (UniqueName: \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\") pod \"78494d89-3baa-4009-a153-aefff1ba4597\" (UID: \"78494d89-3baa-4009-a153-aefff1ba4597\") " Oct 30 09:31:53.996211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.996176 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca" pod "78494d89-3baa-4009-a153-aefff1ba4597" (UID: "78494d89-3baa-4009-a153-aefff1ba4597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:53.997849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.997822 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca" pod "78494d89-3baa-4009-a153-aefff1ba4597" (UID: "78494d89-3baa-4009-a153-aefff1ba4597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:53.998092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:53.998068 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf" pod "78494d89-3baa-4009-a153-aefff1ba4597" (UID: "78494d89-3baa-4009-a153-aefff1ba4597"). InnerVolumeSpecName "kube-api-access-gxwcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:54.097318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.097281 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:54.097318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.097313 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwcf\" (UniqueName: \"kubernetes.io/projected/78494d89-3baa-4009-a153-aefff1ba4597-kube-api-access-gxwcf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:54.097318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.097323 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78494d89-3baa-4009-a153-aefff1ba4597-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:54.710804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.710719 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" Oct 30 09:31:54.710804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.710731 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44" event={"ID":"78494d89-3baa-4009-a153-aefff1ba4597","Type":"ContainerDied","Data":"7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd"} Oct 30 09:31:54.710804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.710782 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7456cd0628735142ea9b78bb4aba33c4b7bf3833dbfa8ac8d40261d8623b42fd" Oct 30 09:31:54.767313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:54.767273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.228535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.228478 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.324610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.324576 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.427578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.427549 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.687832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.687791 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.960782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.960688 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:55.968500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.968452 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9"] Oct 30 09:31:55.974372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:55.974334 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eff9fd-f6xg9"] Oct 30 09:31:56.052049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:56.052016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:56.401173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:56.401138 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86" path="/var/lib/kubelet/pods/36b78f4c-9e32-4b5a-a67e-3fbb9dc67e86/volumes" Oct 30 09:31:56.476597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:56.476556 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:57.670351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:57.670314 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:57.807261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:57.807195 4920 generic.go:356] "Generic (PLEG): container finished" podID="a703116f-bf24-46d0-a530-533f2022386f" containerID="58d366b7d38f6081ac205cf8fbc3be66faa0def689e13aad6c49f95651e5df5e" exitCode=0 Oct 30 09:31:57.807424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:57.807279 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerDied","Data":"58d366b7d38f6081ac205cf8fbc3be66faa0def689e13aad6c49f95651e5df5e"} Oct 30 09:31:57.807495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:57.807408 4920 scope.go:122] "RemoveContainer" containerID="16bcffef5b7c15b2eee86b164fafd7f259b88f5ebc0f24044fe754f413726c15" Oct 30 09:31:58.171680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.170989 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:31:58.285787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.285737 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.285993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.285836 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.285993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.285882 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.285993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.285925 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.285993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.285954 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.286251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286097 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.286251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286114 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:58.286251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286195 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:58.286423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286283 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.286423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286351 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.286423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286390 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\") pod \"a703116f-bf24-46d0-a530-533f2022386f\" (UID: \"a703116f-bf24-46d0-a530-533f2022386f\") " Oct 30 09:31:58.286604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286420 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:58.286663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.286619 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:58.287067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.287042 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.287233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.287071 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.287233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.287088 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.287233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.287103 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.287552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.287521 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:58.288630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.288598 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:58.288630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.288605 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:58.288763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.288634 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "kube-api-access-wlh4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:58.289055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.289038 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo" pod "a703116f-bf24-46d0-a530-533f2022386f" (UID: "a703116f-bf24-46d0-a530-533f2022386f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:31:58.388909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.388827 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a703116f-bf24-46d0-a530-533f2022386f-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.388909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.388883 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a703116f-bf24-46d0-a530-533f2022386f-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.388909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.388893 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a703116f-bf24-46d0-a530-533f2022386f-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.388909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.388904 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.388909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.388913 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlh4b\" (UniqueName: \"kubernetes.io/projected/a703116f-bf24-46d0-a530-533f2022386f-kube-api-access-wlh4b\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:31:58.819516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.819483 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" Oct 30 09:31:58.819516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.819500 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh" event={"ID":"a703116f-bf24-46d0-a530-533f2022386f","Type":"ContainerDied","Data":"662bf8a7cf48f0068911141d3f296a86799384fd756c58b629630186a7a8b563"} Oct 30 09:31:58.820068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.819566 4920 scope.go:122] "RemoveContainer" containerID="58d366b7d38f6081ac205cf8fbc3be66faa0def689e13aad6c49f95651e5df5e" Oct 30 09:31:58.871764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.871727 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:31:58.883422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:58.883386 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d95d35-openshift-default-7684d6f869-9bkhh"] Oct 30 09:31:59.304643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:59.304606 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:31:59.423313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:31:59.423264 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:00.399320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:00.399285 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a703116f-bf24-46d0-a530-533f2022386f" path="/var/lib/kubelet/pods/a703116f-bf24-46d0-a530-533f2022386f/volumes" Oct 30 09:32:01.110052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:01.110011 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:01.145411 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:32:01.145371 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:32:01.305480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:01.305441 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:01.584110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:01.584071 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:02.297782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.297742 4920 scope.go:122] "RemoveContainer" containerID="8c5cab9ef66cfcddf3df14083f57635edcd57bd18f524514b3b8a765a3ec1b84" Oct 30 09:32:02.311515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.311491 4920 scope.go:122] "RemoveContainer" containerID="3c9f2995fbf48255cbafee87ae048ee3bdba05c912213834d876f7643fd9231e" Oct 30 09:32:02.326008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.325980 4920 scope.go:122] "RemoveContainer" containerID="c30c7a4bce1baf59022a7fe044e9098d2cfc915f307be5097f10d4204e93543a" Oct 30 09:32:02.336521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.336501 4920 scope.go:122] "RemoveContainer" containerID="fc83b96013f548945e90fd8ca9a618655413a11e69b3a9896c04e469f5418075" Oct 30 09:32:02.346897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.346873 4920 scope.go:122] "RemoveContainer" containerID="d23b9418dcd6bde52f7745fb907ebc86c94f1dbb02c0c1fd21b22f215f0582f1" Oct 30 09:32:02.356608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:02.356576 4920 scope.go:122] "RemoveContainer" containerID="77e3a29e3cf2024f10ba58743a9573135cd257e044f5ccf7f5dede7924786de1" Oct 30 09:32:03.199773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:03.199734 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:03.348240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:03.342787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:03.699462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:03.699424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:05.112243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:05.111684 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:06.402883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:06.402642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.168688 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb"] Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169736 4920 cpu_manager.go:414] "containerMap: removing container" podUID="78494d89-3baa-4009-a153-aefff1ba4597" containerName="revoke-keys" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169759 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78494d89-3baa-4009-a153-aefff1ba4597" containerName="revoke-keys" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169844 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169854 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169870 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.169880 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.170047 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.170062 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a703116f-bf24-46d0-a530-533f2022386f" containerName="istio-proxy" Oct 30 09:32:07.171308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.170078 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="78494d89-3baa-4009-a153-aefff1ba4597" containerName="revoke-keys" Oct 30 09:32:07.176841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.176813 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.185759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.185728 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb"] Oct 30 09:32:07.193198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.193171 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:32:07.204478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.204449 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:32:07.325533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.325481 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rgv\" (UniqueName: \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.325726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.325706 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.325889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.325871 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.427210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.427112 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.427210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.427241 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.427805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.427300 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rgv\" (UniqueName: \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.427868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.427839 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.429665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.429639 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.447792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.447768 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rgv\" (UniqueName: \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\") pod \"maas-api-revoke-sub-d55f4f86-pmrzb\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.507717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.507684 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:07.762729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.762653 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:07.813876 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:07.813833 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaecb335_0acc_41a4_80ed_360fba9f1523.slice/crio-c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb WatchSource:0}: Error finding container c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb: Status 404 returned error can't find the container with id c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb Oct 30 09:32:07.846844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.818595 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb"] Oct 30 09:32:07.958907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:07.958874 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" event={"ID":"faecb335-0acc-41a4-80ed-360fba9f1523","Type":"ContainerStarted","Data":"c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb"} Oct 30 09:32:08.967629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:08.967595 4920 generic.go:356] "Generic (PLEG): container finished" podID="faecb335-0acc-41a4-80ed-360fba9f1523" containerID="5a8a65d5356fda01d7b3d07b82db1d10b169327f25f5c6e8fbf35b6f83d45757" exitCode=0 Oct 30 09:32:08.968033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:08.967677 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" event={"ID":"faecb335-0acc-41a4-80ed-360fba9f1523","Type":"ContainerDied","Data":"5a8a65d5356fda01d7b3d07b82db1d10b169327f25f5c6e8fbf35b6f83d45757"} Oct 30 09:32:09.498198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:09.498153 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:10.149965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.149935 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:10.170706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.170679 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\" (UniqueName: \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\") pod \"faecb335-0acc-41a4-80ed-360fba9f1523\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " Oct 30 09:32:10.170870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.170796 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\") pod \"faecb335-0acc-41a4-80ed-360fba9f1523\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " Oct 30 09:32:10.170870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.170834 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\") pod \"faecb335-0acc-41a4-80ed-360fba9f1523\" (UID: \"faecb335-0acc-41a4-80ed-360fba9f1523\") " Oct 30 09:32:10.171193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.171147 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca" pod "faecb335-0acc-41a4-80ed-360fba9f1523" (UID: "faecb335-0acc-41a4-80ed-360fba9f1523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:10.172727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.172704 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv" pod "faecb335-0acc-41a4-80ed-360fba9f1523" (UID: "faecb335-0acc-41a4-80ed-360fba9f1523"). InnerVolumeSpecName "kube-api-access-29rgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:10.172837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.172821 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca" pod "faecb335-0acc-41a4-80ed-360fba9f1523" (UID: "faecb335-0acc-41a4-80ed-360fba9f1523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:10.272801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.272717 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:10.272801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.272749 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faecb335-0acc-41a4-80ed-360fba9f1523-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:10.272801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.272759 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rgv\" (UniqueName: \"kubernetes.io/projected/faecb335-0acc-41a4-80ed-360fba9f1523-kube-api-access-29rgv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:10.982959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.982930 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" Oct 30 09:32:10.983156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.983018 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb" event={"ID":"faecb335-0acc-41a4-80ed-360fba9f1523","Type":"ContainerDied","Data":"c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb"} Oct 30 09:32:10.983156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:10.983078 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d206d2a69ae53dca024f58f8d6608ece1b8e80e6612aa958d401144e1009eb" Oct 30 09:32:11.541990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:11.541953 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:12.673050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:12.673012 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:13.227052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:13.227013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:13.378237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:13.376551 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:13.480518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:13.479495 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:14.829507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:14.829470 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:14.941839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:14.941801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:15.063097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:15.063061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:15.179048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:15.178916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:15.276944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:15.276903 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:16.692304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:16.692266 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:16.810265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:16.810208 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:16.974747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:16.974636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:18.194421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:18.194351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:18.616773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:18.616736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:18.786801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:18.786758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:18.928296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:18.921752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:19.053603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:19.053565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:20.240476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.239534 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:32:20.242052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.241394 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" podUID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" containerName="maas-api" containerID="cri-o://83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236" gracePeriod=30 Oct 30 09:32:20.294359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.294284 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:20.541300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.541274 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:32:20.673241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.660419 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\") pod \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " Oct 30 09:32:20.673241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.660525 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\" (UniqueName: \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\") pod \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " Oct 30 09:32:20.673241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.660566 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\") pod \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " Oct 30 09:32:20.673241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.660757 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\") pod \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\" (UID: \"eaea09ae-2903-4dbc-bbc7-2583ad7f2508\") " Oct 30 09:32:20.697457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.686803 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca" pod "eaea09ae-2903-4dbc-bbc7-2583ad7f2508" (UID: "eaea09ae-2903-4dbc-bbc7-2583ad7f2508"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:20.697457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.692403 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2" pod "eaea09ae-2903-4dbc-bbc7-2583ad7f2508" (UID: "eaea09ae-2903-4dbc-bbc7-2583ad7f2508"). InnerVolumeSpecName "kube-api-access-4bnq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:20.697457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.693970 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs" pod "eaea09ae-2903-4dbc-bbc7-2583ad7f2508" (UID: "eaea09ae-2903-4dbc-bbc7-2583ad7f2508"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:20.697457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.696956 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls" pod "eaea09ae-2903-4dbc-bbc7-2583ad7f2508" (UID: "eaea09ae-2903-4dbc-bbc7-2583ad7f2508"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:20.784331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.775741 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.784331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.777204 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.784331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.777241 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.784331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.777257 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnq2\" (UniqueName: \"kubernetes.io/projected/eaea09ae-2903-4dbc-bbc7-2583ad7f2508-kube-api-access-4bnq2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:20.808504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.801692 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:20.947361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:20.947330 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:21.077063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.076980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:21.099135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.099096 4920 generic.go:356] "Generic (PLEG): container finished" podID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" containerID="83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236" exitCode=0 Oct 30 09:32:21.099367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.099159 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" Oct 30 09:32:21.099367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.099313 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" event={"ID":"eaea09ae-2903-4dbc-bbc7-2583ad7f2508","Type":"ContainerDied","Data":"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236"} Oct 30 09:32:21.099675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.099650 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85" event={"ID":"eaea09ae-2903-4dbc-bbc7-2583ad7f2508","Type":"ContainerDied","Data":"4e24d63ed215447dd6261671a9075c0b3a7775522337a686cad151e25ac49418"} Oct 30 09:32:21.099757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.099695 4920 scope.go:122] "RemoveContainer" containerID="83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236" Oct 30 09:32:21.113863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.113829 4920 scope.go:122] "RemoveContainer" containerID="83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236" Oct 30 09:32:21.114285 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:32:21.114207 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236\": container with ID starting with 83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236 not found: ID does not exist" containerID="83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236" Oct 30 09:32:21.114373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.114300 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236"} err="failed to get container status \"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236\": rpc error: code = NotFound desc = could not find container \"83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236\": container with ID starting with 83a8abc4518ca2f80981a0740b345364089aed6f0df414efebe8675ba3466236 not found: ID does not exist" Oct 30 09:32:21.172335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.169196 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:32:21.179143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.179104 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-066f95-65c4bbc4db-l6d85"] Oct 30 09:32:21.494234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.494134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:21.748739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.748647 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:32:21.749451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749429 4920 cpu_manager.go:414] "containerMap: removing container" podUID="faecb335-0acc-41a4-80ed-360fba9f1523" containerName="revoke-keys" Oct 30 09:32:21.749537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749453 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faecb335-0acc-41a4-80ed-360fba9f1523" containerName="revoke-keys" Oct 30 09:32:21.749537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749478 4920 cpu_manager.go:414] "containerMap: removing container" podUID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" containerName="maas-api" Oct 30 09:32:21.749537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749485 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" containerName="maas-api" Oct 30 09:32:21.749699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749651 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" containerName="maas-api" Oct 30 09:32:21.749699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.749681 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="faecb335-0acc-41a4-80ed-360fba9f1523" containerName="revoke-keys" Oct 30 09:32:21.754667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.754644 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.788133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.788099 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:32:21.790669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.790635 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.790848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.790786 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtgh\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.790848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.790834 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.790977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.790944 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.791036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.791008 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.791101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.791071 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.791157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.791106 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.791267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.791175 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.791400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.791380 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892198 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtgh\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892268 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892311 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892348 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892376 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892409 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892462 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892579 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892698 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892720 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892895 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.892959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.892951 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.893087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.893061 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.893336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.893318 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.894829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.894797 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.894990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.894975 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.914312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.914278 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtgh\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:21.914451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:21.914362 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:22.076116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.076082 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:22.399914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.399837 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaea09ae-2903-4dbc-bbc7-2583ad7f2508" path="/var/lib/kubelet/pods/eaea09ae-2903-4dbc-bbc7-2583ad7f2508/volumes" Oct 30 09:32:22.420451 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:22.420414 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780c1b94_75c2_4585_b657_08f3d0481de4.slice/crio-6f20cc8e8eed940484252a58c62cfc013c4a9ef08b4bb23189305580adf20a21 WatchSource:0}: Error finding container 6f20cc8e8eed940484252a58c62cfc013c4a9ef08b4bb23189305580adf20a21: Status 404 returned error can't find the container with id 6f20cc8e8eed940484252a58c62cfc013c4a9ef08b4bb23189305580adf20a21 Oct 30 09:32:22.422130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.422108 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:32:22.422970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.422939 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:22.423053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.423004 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:22.423053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.423034 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:22.840611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.840567 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:22.974344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:22.974186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:23.079371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:23.079335 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:23.375966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:23.375921 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" event={"ID":"780c1b94-75c2-4585-b657-08f3d0481de4","Type":"ContainerStarted","Data":"6106f1ccf6806f1bbcfdbbb58f3d49839f2e603fbbaf8cd78f29466b126a1644"} Oct 30 09:32:23.376173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:23.376003 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" event={"ID":"780c1b94-75c2-4585-b657-08f3d0481de4","Type":"ContainerStarted","Data":"6f20cc8e8eed940484252a58c62cfc013c4a9ef08b4bb23189305580adf20a21"} Oct 30 09:32:23.444239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:23.444039 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podStartSLOduration=2.444024711 podStartE2EDuration="2.444024711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:23.44044591 +0000 UTC m=+2307.163873080" watchObservedRunningTime="2026-10-30 09:32:23.444024711 +0000 UTC m=+2307.167451911" Oct 30 09:32:24.076196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:24.076155 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:24.078095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:24.078066 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:24.078195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:24.078132 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:24.679681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:24.679647 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:24.835725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:24.835690 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:25.112267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:25.099431 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:25.112267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:25.099524 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:25.186056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:25.186023 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:26.077282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:26.077247 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:26.077473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:26.077302 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:27.076998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.076964 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:27.077467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.077030 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:27.357828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.357740 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:27.524673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.524595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:27.627739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.627600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:27.870724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:27.870666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:28.080184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.080150 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:28.080907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.080875 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:28.708288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.697960 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:32:28.708288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.700699 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" containerName="maas-api" containerID="cri-o://dc8c2747d8065c055cd90701a3284cac9a9107f34abc5fdfcbf9c0a34fd84bc1" gracePeriod=30 Oct 30 09:32:28.734943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.734877 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.191:8443/health\": dial tcp 10.132.0.191:8443: connect: connection refused" Oct 30 09:32:28.748559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.748528 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:28.854256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.854144 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:32:28.854546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.854379 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" podUID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" containerName="payload-processing" containerID="cri-o://9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b" gracePeriod=30 Oct 30 09:32:28.873809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.873776 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:32:28.874203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.874147 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" podUID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" containerName="payload-pre-processing" containerID="cri-o://986988b686842b906904407a194c953bdf34f7736ce8d183f46d49b3ae2da458" gracePeriod=30 Oct 30 09:32:28.927211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:28.927165 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:29.046331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.021316 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:29.078597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.078561 4920 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 09:32:29.078739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.078620 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 09:32:29.298834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.297777 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:29.380936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.380900 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:32:29.422514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.422476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:29.490800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.490715 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44"] Oct 30 09:32:29.499587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.499279 4920 generic.go:356] "Generic (PLEG): container finished" podID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" containerID="986988b686842b906904407a194c953bdf34f7736ce8d183f46d49b3ae2da458" exitCode=0 Oct 30 09:32:29.500004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.499410 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" event={"ID":"6d8d9b27-63b9-4ea4-a889-b732ce46498e","Type":"ContainerDied","Data":"986988b686842b906904407a194c953bdf34f7736ce8d183f46d49b3ae2da458"} Oct 30 09:32:29.565315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.514359 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\" (UniqueName: \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\") pod \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " Oct 30 09:32:29.567470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.549821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:29.567933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.567912 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\") pod \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\" (UID: \"c1c46a36-4a8d-4db2-bc02-be1e87f52b13\") " Oct 30 09:32:29.569338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.568436 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume" pod "c1c46a36-4a8d-4db2-bc02-be1e87f52b13" (UID: "c1c46a36-4a8d-4db2-bc02-be1e87f52b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:29.570584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.570563 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.573659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.573634 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d87c85-bqq44"] Oct 30 09:32:29.595255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.579619 4920 generic.go:356] "Generic (PLEG): container finished" podID="423a28f5-0f1b-4718-9126-d94482895e2d" containerID="dc8c2747d8065c055cd90701a3284cac9a9107f34abc5fdfcbf9c0a34fd84bc1" exitCode=0 Oct 30 09:32:29.595255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.579856 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" event={"ID":"423a28f5-0f1b-4718-9126-d94482895e2d","Type":"ContainerDied","Data":"dc8c2747d8065c055cd90701a3284cac9a9107f34abc5fdfcbf9c0a34fd84bc1"} Oct 30 09:32:29.595255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.584008 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh" pod "c1c46a36-4a8d-4db2-bc02-be1e87f52b13" (UID: "c1c46a36-4a8d-4db2-bc02-be1e87f52b13"). InnerVolumeSpecName "kube-api-access-cp7gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:29.595255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.588795 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:32:29.605704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.605672 4920 generic.go:356] "Generic (PLEG): container finished" podID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" containerID="9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b" exitCode=0 Oct 30 09:32:29.605979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.605773 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" Oct 30 09:32:29.606823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.606797 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" event={"ID":"c1c46a36-4a8d-4db2-bc02-be1e87f52b13","Type":"ContainerDied","Data":"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b"} Oct 30 09:32:29.606929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.606855 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6" event={"ID":"c1c46a36-4a8d-4db2-bc02-be1e87f52b13","Type":"ContainerDied","Data":"4586b2895689f0d911eaaa760ec8e6ffcf6af9dd9c0b4c689e1de007d7630682"} Oct 30 09:32:29.606929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.606884 4920 scope.go:122] "RemoveContainer" containerID="9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b" Oct 30 09:32:29.615699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.615617 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:32:29.634173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.634073 4920 scope.go:122] "RemoveContainer" containerID="9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b" Oct 30 09:32:29.634646 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:32:29.634579 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b\": container with ID starting with 9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b not found: ID does not exist" containerID="9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b" Oct 30 09:32:29.634646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.634614 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b"} err="failed to get container status \"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b\": rpc error: code = NotFound desc = could not find container \"9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b\": container with ID starting with 9a4aca1b51ae1a9904eee8db3aab9c80693641769c1d2b62287792d0afb8a69b not found: ID does not exist" Oct 30 09:32:29.671677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671639 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\") pod \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " Oct 30 09:32:29.671877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671723 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\") pod \"423a28f5-0f1b-4718-9126-d94482895e2d\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " Oct 30 09:32:29.671877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671760 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\" (UniqueName: \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\") pod \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\" (UID: \"6d8d9b27-63b9-4ea4-a889-b732ce46498e\") " Oct 30 09:32:29.671981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671872 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") pod \"423a28f5-0f1b-4718-9126-d94482895e2d\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " Oct 30 09:32:29.671981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671942 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\" (UniqueName: \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\") pod \"423a28f5-0f1b-4718-9126-d94482895e2d\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " Oct 30 09:32:29.672058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.671986 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\") pod \"423a28f5-0f1b-4718-9126-d94482895e2d\" (UID: \"423a28f5-0f1b-4718-9126-d94482895e2d\") " Oct 30 09:32:29.672479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.672448 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp7gh\" (UniqueName: \"kubernetes.io/projected/c1c46a36-4a8d-4db2-bc02-be1e87f52b13-kube-api-access-cp7gh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.673262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.673237 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume" pod "6d8d9b27-63b9-4ea4-a889-b732ce46498e" (UID: "6d8d9b27-63b9-4ea4-a889-b732ce46498e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:29.674735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.674699 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca" pod "423a28f5-0f1b-4718-9126-d94482895e2d" (UID: "423a28f5-0f1b-4718-9126-d94482895e2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:29.676765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.676722 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g" pod "6d8d9b27-63b9-4ea4-a889-b732ce46498e" (UID: "6d8d9b27-63b9-4ea4-a889-b732ce46498e"). InnerVolumeSpecName "kube-api-access-vzt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:29.676952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.676925 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg" pod "423a28f5-0f1b-4718-9126-d94482895e2d" (UID: "423a28f5-0f1b-4718-9126-d94482895e2d"). InnerVolumeSpecName "kube-api-access-vbqpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:29.677151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.677122 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs" pod "423a28f5-0f1b-4718-9126-d94482895e2d" (UID: "423a28f5-0f1b-4718-9126-d94482895e2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:29.677674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.677649 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls" pod "423a28f5-0f1b-4718-9126-d94482895e2d" (UID: "423a28f5-0f1b-4718-9126-d94482895e2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:29.724803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.723193 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:32:29.731186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.731149 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-6bfc9c74dd-wrpj6"] Oct 30 09:32:29.773613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773574 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzt4g\" (UniqueName: \"kubernetes.io/projected/6d8d9b27-63b9-4ea4-a889-b732ce46498e-kube-api-access-vzt4g\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.773613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773616 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.773830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773634 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbqpg\" (UniqueName: \"kubernetes.io/projected/423a28f5-0f1b-4718-9126-d94482895e2d-kube-api-access-vbqpg\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.773830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773651 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a28f5-0f1b-4718-9126-d94482895e2d-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.773830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773666 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8d9b27-63b9-4ea4-a889-b732ce46498e-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.773830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.773679 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a28f5-0f1b-4718-9126-d94482895e2d-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:29.950748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:29.950696 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:30.378790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.378751 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:30.379450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.379428 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:30.380250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.380204 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:32:30.398946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.398912 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78494d89-3baa-4009-a153-aefff1ba4597" path="/var/lib/kubelet/pods/78494d89-3baa-4009-a153-aefff1ba4597/volumes" Oct 30 09:32:30.399379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.399363 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" path="/var/lib/kubelet/pods/c1c46a36-4a8d-4db2-bc02-be1e87f52b13/volumes" Oct 30 09:32:30.615672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.615643 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" Oct 30 09:32:30.615873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.615645 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m" event={"ID":"6d8d9b27-63b9-4ea4-a889-b732ce46498e","Type":"ContainerDied","Data":"7c9fb2e8e6c19845cc249aec39a129d39bec7e5c02e6bb5f3dad8fc9f7a3268c"} Oct 30 09:32:30.615873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.615815 4920 scope.go:122] "RemoveContainer" containerID="986988b686842b906904407a194c953bdf34f7736ce8d183f46d49b3ae2da458" Oct 30 09:32:30.618182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.618144 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" event={"ID":"423a28f5-0f1b-4718-9126-d94482895e2d","Type":"ContainerDied","Data":"871e8dd78db6a79cf48acad3ec01ac03a7923861688e4a3e9a4ec232f0a0cf87"} Oct 30 09:32:30.618510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.618454 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd" Oct 30 09:32:30.628711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.628682 4920 scope.go:122] "RemoveContainer" containerID="dc8c2747d8065c055cd90701a3284cac9a9107f34abc5fdfcbf9c0a34fd84bc1" Oct 30 09:32:30.669134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.669103 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:32:30.672903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.672875 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5584478bcf-6nr6m"] Oct 30 09:32:30.705679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.705647 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:32:30.709299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:30.709272 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d87c85-79bc7779d4-gpfnd"] Oct 30 09:32:31.146007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:31.145955 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:32:31.208852 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:32:31.208814 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:32:31.232954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:31.232923 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:32.467990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:32.467951 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" path="/var/lib/kubelet/pods/423a28f5-0f1b-4718-9126-d94482895e2d/volumes" Oct 30 09:32:32.469442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:32.469412 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" path="/var/lib/kubelet/pods/6d8d9b27-63b9-4ea4-a889-b732ce46498e/volumes" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.334549 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335559 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" containerName="payload-processing" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335582 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" containerName="payload-processing" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335641 4920 cpu_manager.go:414] "containerMap: removing container" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" containerName="maas-api" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335649 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" containerName="maas-api" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335675 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" containerName="payload-pre-processing" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335684 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" containerName="payload-pre-processing" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335850 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="423a28f5-0f1b-4718-9126-d94482895e2d" containerName="maas-api" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335867 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8d9b27-63b9-4ea4-a889-b732ce46498e" containerName="payload-pre-processing" Oct 30 09:32:33.338244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.335884 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c46a36-4a8d-4db2-bc02-be1e87f52b13" containerName="payload-processing" Oct 30 09:32:33.537248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.514072 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:33.619679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.604845 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:33.652239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.640375 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.775368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.751412 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fc7ee7-serving-cert\"" Oct 30 09:32:33.871157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.869368 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.871157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.869633 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.871157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.869707 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.871157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.869770 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnvd\" (UniqueName: \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.884028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.883992 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:32:33.901519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.901477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:33.973685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.973618 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.979510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.979480 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:33.979932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:33.979908 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.001148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.001093 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.002027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.002002 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnvd\" (UniqueName: \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.017045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.017005 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.022600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.022572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.039250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.036970 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnvd\" (UniqueName: \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\") pod \"maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.236541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.236451 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:34.588311 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:34.588270 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf964b5_9081_43a1_8345_f60fb1a375eb.slice/crio-4eb07ed2275e108276d86da08ec8503b9738ee83f7b6377389c6e32a4fbf0d6b WatchSource:0}: Error finding container 4eb07ed2275e108276d86da08ec8503b9738ee83f7b6377389c6e32a4fbf0d6b: Status 404 returned error can't find the container with id 4eb07ed2275e108276d86da08ec8503b9738ee83f7b6377389c6e32a4fbf0d6b Oct 30 09:32:34.590419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.590393 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:32:34.760923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.760889 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:34.874936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.874840 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd"] Oct 30 09:32:34.880376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.880346 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:34.880880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.880846 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd"] Oct 30 09:32:34.894629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.894595 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" event={"ID":"ebf964b5-9081-43a1-8345-f60fb1a375eb","Type":"ContainerStarted","Data":"4eb07ed2275e108276d86da08ec8503b9738ee83f7b6377389c6e32a4fbf0d6b"} Oct 30 09:32:34.895436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.895413 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:32:34.907011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:34.906982 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:32:35.011890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.011851 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:35.040792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.040693 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.040970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.040863 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjmm\" (UniqueName: \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.040970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.040919 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.155288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.142168 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.155288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.142319 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjmm\" (UniqueName: \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.155288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.142383 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.155288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.143329 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.195239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.192327 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.229202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.229163 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:35.229901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.229878 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjmm\" (UniqueName: \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\") pod \"maas-api-revoke-sub-7d23dab0-clmqd\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.501339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.501210 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:35.674527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.674484 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:32:35.680250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.680209 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.709476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.709444 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7c88ae-openshift-default-dockercfg-rffvm\"" Oct 30 09:32:35.715162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.715131 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:32:35.728475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.728444 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:35.753594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753505 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.753594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753561 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.753841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753706 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.753841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753739 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.753970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753857 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.753970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.753891 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.754090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.754011 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.754090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.754045 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7nh\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.754090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.754070 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.874094 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.874164 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7nh\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.874205 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.886334 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.886490 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.886690 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.886821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.886732 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.895084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.895051 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.895310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.895287 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.895401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.895335 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.909243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.895722 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.939746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.935190 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.939746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.935747 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.939746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.936393 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.949875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.949586 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.953891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.898899 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd"] Oct 30 09:32:35.967304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.967265 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:35.989162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.989123 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" event={"ID":"ebf964b5-9081-43a1-8345-f60fb1a375eb","Type":"ContainerStarted","Data":"ca770ee7a68847850a7cfeba8abfed67eb5705ae4912ef4de6c9dc91e9673e82"} Oct 30 09:32:35.990338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.990314 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:35.995049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:35.994928 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" event={"ID":"524df766-3f38-458c-91af-1e07d7950558","Type":"ContainerStarted","Data":"0b3269f5b76c21d19ee1afe7372b335e3c73204c2b9ba5b05bf4a13b9d8fd81f"} Oct 30 09:32:36.027831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.027800 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:36.032721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.032694 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7nh\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\") pod \"e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:36.044901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.044689 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" podStartSLOduration=2.752591531 podStartE2EDuration="3.044666438s" totalImagesPullingTime="292.074907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:36.03802401 +0000 UTC m=+2319.761451154" watchObservedRunningTime="2026-10-30 09:32:36.044666438 +0000 UTC m=+2319.768093583" Oct 30 09:32:36.079416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.079383 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:36.214260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.214209 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:36.310014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.309979 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:36.636945 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:36.636912 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb95ad4_6def_4bba_ba52_d7ba00cd8df3.slice/crio-83d677466bcbe8088dbc50654f3504b2cb44cebac00d1578574020eead6b0e86 WatchSource:0}: Error finding container 83d677466bcbe8088dbc50654f3504b2cb44cebac00d1578574020eead6b0e86: Status 404 returned error can't find the container with id 83d677466bcbe8088dbc50654f3504b2cb44cebac00d1578574020eead6b0e86 Oct 30 09:32:36.638814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.638791 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:32:36.640465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.640424 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:36.640554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.640507 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:36.640599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:36.640550 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:37.006543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.006452 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" event={"ID":"524df766-3f38-458c-91af-1e07d7950558","Type":"ContainerStarted","Data":"b4f46efa6cfdfaab9d7c3b18a2d1f9cb4c2f1a177982f76daac85e116e3f55ed"} Oct 30 09:32:37.008693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.008661 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerStarted","Data":"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587"} Oct 30 09:32:37.008837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.008718 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerStarted","Data":"83d677466bcbe8088dbc50654f3504b2cb44cebac00d1578574020eead6b0e86"} Oct 30 09:32:37.047281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.047233 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" podStartSLOduration=3.047205522 podStartE2EDuration="3.047205522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:37.044435785 +0000 UTC m=+2320.767862965" watchObservedRunningTime="2026-10-30 09:32:37.047205522 +0000 UTC m=+2320.770632665" Oct 30 09:32:37.089167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.089108 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podStartSLOduration=2.08909343 podStartE2EDuration="2.08909343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:37.08534754 +0000 UTC m=+2320.808774682" watchObservedRunningTime="2026-10-30 09:32:37.08909343 +0000 UTC m=+2320.812520573" Oct 30 09:32:37.318245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.318000 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:37.325210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.325170 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:37.694122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:37.694044 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:38.019975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.019878 4920 generic.go:356] "Generic (PLEG): container finished" podID="524df766-3f38-458c-91af-1e07d7950558" containerID="b4f46efa6cfdfaab9d7c3b18a2d1f9cb4c2f1a177982f76daac85e116e3f55ed" exitCode=0 Oct 30 09:32:38.020385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.019965 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" event={"ID":"524df766-3f38-458c-91af-1e07d7950558","Type":"ContainerDied","Data":"b4f46efa6cfdfaab9d7c3b18a2d1f9cb4c2f1a177982f76daac85e116e3f55ed"} Oct 30 09:32:38.020557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.020538 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:38.021496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.021472 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:32:38.405085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.405049 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:32:38.405342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:38.405233 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" containerID="cri-o://993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54" gracePeriod=30 Oct 30 09:32:39.193655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.193621 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:39.255295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.255255 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\") pod \"524df766-3f38-458c-91af-1e07d7950558\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " Oct 30 09:32:39.255498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.255392 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\") pod \"524df766-3f38-458c-91af-1e07d7950558\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " Oct 30 09:32:39.255608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.255588 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\" (UniqueName: \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\") pod \"524df766-3f38-458c-91af-1e07d7950558\" (UID: \"524df766-3f38-458c-91af-1e07d7950558\") " Oct 30 09:32:39.255851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.255827 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca" pod "524df766-3f38-458c-91af-1e07d7950558" (UID: "524df766-3f38-458c-91af-1e07d7950558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:39.256134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.256114 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524df766-3f38-458c-91af-1e07d7950558-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:39.257474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.257452 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca" pod "524df766-3f38-458c-91af-1e07d7950558" (UID: "524df766-3f38-458c-91af-1e07d7950558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:39.258125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.258108 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm" pod "524df766-3f38-458c-91af-1e07d7950558" (UID: "524df766-3f38-458c-91af-1e07d7950558"). InnerVolumeSpecName "kube-api-access-fpjmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:39.357421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.357385 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524df766-3f38-458c-91af-1e07d7950558-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:39.357421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.357415 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjmm\" (UniqueName: \"kubernetes.io/projected/524df766-3f38-458c-91af-1e07d7950558-kube-api-access-fpjmm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:39.425250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.425168 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:39.534241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.533698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:39.635877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:39.635761 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:40.039438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:40.039408 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" Oct 30 09:32:40.039672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:40.039415 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd" event={"ID":"524df766-3f38-458c-91af-1e07d7950558","Type":"ContainerDied","Data":"0b3269f5b76c21d19ee1afe7372b335e3c73204c2b9ba5b05bf4a13b9d8fd81f"} Oct 30 09:32:40.039672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:40.039547 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3269f5b76c21d19ee1afe7372b335e3c73204c2b9ba5b05bf4a13b9d8fd81f" Oct 30 09:32:41.120101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:41.120010 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:41.313184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:41.313151 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:41.411309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:41.411192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:42.882871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:42.882838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:43.027583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.027550 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:32:43.690458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.690405 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:32:43.806168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806135 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806198 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806288 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806330 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806401 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806521 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806586 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806612 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.806687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806650 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806718 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\") pod \"61562d3d-8726-4942-a7a6-37190abd9f11\" (UID: \"61562d3d-8726-4942-a7a6-37190abd9f11\") " Oct 30 09:32:43.806864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806788 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.806968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806869 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.806968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.806920 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.807211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.807184 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:43.807499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.807480 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.807613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.807601 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.807705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.807693 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61562d3d-8726-4942-a7a6-37190abd9f11-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.808887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.808864 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.809022 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.808894 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.809351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.809315 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:43.809483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.809447 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "kube-api-access-766kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:43.809592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.809576 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:32:43.809952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.809929 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token" pod "61562d3d-8726-4942-a7a6-37190abd9f11" (UID: "61562d3d-8726-4942-a7a6-37190abd9f11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:43.910251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.910196 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.910251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.910249 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766kk\" (UniqueName: \"kubernetes.io/projected/61562d3d-8726-4942-a7a6-37190abd9f11-kube-api-access-766kk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.910721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.910265 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61562d3d-8726-4942-a7a6-37190abd9f11-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:43.910721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:43.910282 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61562d3d-8726-4942-a7a6-37190abd9f11-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:32:44.076537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.076492 4920 generic.go:356] "Generic (PLEG): container finished" podID="61562d3d-8726-4942-a7a6-37190abd9f11" containerID="993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54" exitCode=0 Oct 30 09:32:44.076537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.076537 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" Oct 30 09:32:44.076802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.076617 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" event={"ID":"61562d3d-8726-4942-a7a6-37190abd9f11","Type":"ContainerDied","Data":"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54"} Oct 30 09:32:44.076802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.076673 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q" event={"ID":"61562d3d-8726-4942-a7a6-37190abd9f11","Type":"ContainerDied","Data":"64a06ad72554e35273d8905d5b21f53854b2304d944d0f28947ceafa9591a8b6"} Oct 30 09:32:44.076802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.076702 4920 scope.go:122] "RemoveContainer" containerID="993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54" Oct 30 09:32:44.091391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.091374 4920 scope.go:122] "RemoveContainer" containerID="993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54" Oct 30 09:32:44.091712 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:32:44.091692 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54\": container with ID starting with 993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54 not found: ID does not exist" containerID="993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54" Oct 30 09:32:44.091789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.091721 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54"} err="failed to get container status \"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54\": rpc error: code = NotFound desc = could not find container \"993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54\": container with ID starting with 993ad01451cd356dbaa7914c5c366bbcf2315dc00efc1fddc6395ce2d7f2fa54 not found: ID does not exist" Oct 30 09:32:44.133067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.133018 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:32:44.136674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.136645 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5cq5q"] Oct 30 09:32:44.400140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.400059 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" path="/var/lib/kubelet/pods/61562d3d-8726-4942-a7a6-37190abd9f11/volumes" Oct 30 09:32:44.632966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:44.632936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:46.157680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.157644 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:46.591064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.591025 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:32:46.591973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.591947 4920 cpu_manager.go:414] "containerMap: removing container" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" Oct 30 09:32:46.591973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.591971 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" Oct 30 09:32:46.592205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.591987 4920 cpu_manager.go:414] "containerMap: removing container" podUID="524df766-3f38-458c-91af-1e07d7950558" containerName="revoke-keys" Oct 30 09:32:46.592205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.591993 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524df766-3f38-458c-91af-1e07d7950558" containerName="revoke-keys" Oct 30 09:32:46.592205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.592112 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="61562d3d-8726-4942-a7a6-37190abd9f11" containerName="istio-proxy" Oct 30 09:32:46.592205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.592128 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="524df766-3f38-458c-91af-1e07d7950558" containerName="revoke-keys" Oct 30 09:32:46.597497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.597472 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.619752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.616574 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-304459-openshift-default-dockercfg-8w4bn\"" Oct 30 09:32:46.633708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.633676 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:32:46.647830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.647787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:46.650140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650114 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650168 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650307 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650430 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvm6\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650490 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650534 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650594 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650629 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.650677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.650651 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.753536 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvm6\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.753599 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.754855 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755160 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.754008 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755273 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755312 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755350 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755470 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755602 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.755971 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.757241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.757255 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.758070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.757490 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.775240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.773853 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvm6\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.791358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.777794 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.791358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.778198 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.822272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.817739 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:46.822272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.818287 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:46.907292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.906813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:46.927244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:46.924505 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:47.283021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:47.282990 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:32:47.283436 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:47.283332 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2947718f_2e49_48c9_abf1_0a224f149c61.slice/crio-02d53aeabb1d5c62d201de837fbd9c353da2f51a25c4a568e69d40266eeadaaf WatchSource:0}: Error finding container 02d53aeabb1d5c62d201de837fbd9c353da2f51a25c4a568e69d40266eeadaaf: Status 404 returned error can't find the container with id 02d53aeabb1d5c62d201de837fbd9c353da2f51a25c4a568e69d40266eeadaaf Oct 30 09:32:47.286023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:47.285984 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:47.286156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:47.286052 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:47.286156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:47.286081 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:47.597208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:47.597135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:48.111898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.111863 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerStarted","Data":"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29"} Oct 30 09:32:48.112096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.111915 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerStarted","Data":"02d53aeabb1d5c62d201de837fbd9c353da2f51a25c4a568e69d40266eeadaaf"} Oct 30 09:32:48.175621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.175563 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podStartSLOduration=2.175546799 podStartE2EDuration="2.175546799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:48.174751459 +0000 UTC m=+2331.898178612" watchObservedRunningTime="2026-10-30 09:32:48.175546799 +0000 UTC m=+2331.898973942" Oct 30 09:32:48.696004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.695972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:48.925404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.925367 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:48.931308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:48.931284 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:49.121809 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:49.121773 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:49.122903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:49.122883 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:32:50.474114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:50.474074 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:50.583494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:50.583455 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:50.687922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:50.687884 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:50.824824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:50.824786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:52.236435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:52.236390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:52.344394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:52.344359 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:52.446093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:52.446038 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:53.883281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:53.883244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:55.486181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:55.486141 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:56.018770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:56.018721 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:32:57.326896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.326860 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:32:57.332934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.332904 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.379992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.379957 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-7c88ae-serving-cert\"" Oct 30 09:32:57.384329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.384297 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:32:57.385947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.385880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:57.434043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.434003 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvdk\" (UniqueName: \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.434043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.434044 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.434358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.434149 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.434358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.434330 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.490429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.490392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:57.535539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.535503 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvdk\" (UniqueName: \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.535733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.535545 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.535733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.535600 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.536157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.535910 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.536510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.536482 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.538246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.538205 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.538320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.538283 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.554071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.554045 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvdk\" (UniqueName: \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\") pod \"maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.596182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.594933 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:57.695453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.695424 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:57.817793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:57.817747 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:58.000669 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:32:58.000641 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000155aa_ce40_4e39_a5b6_6961b01e88d1.slice/crio-4cdf33730b78052f15d8cada345eac4879c6f6415e2e449c892dd4b879f0dfa1 WatchSource:0}: Error finding container 4cdf33730b78052f15d8cada345eac4879c6f6415e2e449c892dd4b879f0dfa1: Status 404 returned error can't find the container with id 4cdf33730b78052f15d8cada345eac4879c6f6415e2e449c892dd4b879f0dfa1 Oct 30 09:32:58.001997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:58.001971 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:32:58.254558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:58.254467 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" event={"ID":"000155aa-ce40-4e39-a5b6-6961b01e88d1","Type":"ContainerStarted","Data":"4cdf33730b78052f15d8cada345eac4879c6f6415e2e449c892dd4b879f0dfa1"} Oct 30 09:32:58.305692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:58.305648 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:58.437034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:58.437001 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:59.160760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:59.160730 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:32:59.264286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:59.264244 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" event={"ID":"000155aa-ce40-4e39-a5b6-6961b01e88d1","Type":"ContainerStarted","Data":"285413df0db25c7952a2d1046055c7e41369e18849061868073fcb915b8c5f82"} Oct 30 09:32:59.264496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:59.264316 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:32:59.345631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:32:59.332790 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" podStartSLOduration=1.95001044 podStartE2EDuration="2.332765121s" totalImagesPullingTime="382.754681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:59.311022482 +0000 UTC m=+2343.034449653" watchObservedRunningTime="2026-10-30 09:32:59.332765121 +0000 UTC m=+2343.056192265" Oct 30 09:33:00.866943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:00.866912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:01.139090 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:33:01.138994 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:33:01.587156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:01.587124 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:01.692971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:01.692935 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:02.427911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:02.427873 4920 scope.go:122] "RemoveContainer" containerID="8fa49b40fc4ed5d7dc5b6c46f71a6fc708c8084ba55934aab83150bbc8b0006e" Oct 30 09:33:02.556273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:02.556212 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:03.423630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:03.423594 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:03.905874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:03.905834 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:04.359263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.356965 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:33:04.362992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.362966 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.402987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.402954 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:04.408261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.408230 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:33:04.452555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.452527 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-0dc148-openshift-default-dockercfg-gt4wg\"" Oct 30 09:33:04.458660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458628 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.458832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458678 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.458832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458763 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.458942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458878 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.458942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458910 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.458942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.458928 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97m9s\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.459097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.459000 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.459097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.459021 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.459097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.459059 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.504055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.504024 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:04.560080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560047 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560086 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97m9s\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560123 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560144 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560164 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560335 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560386 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560412 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560509 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560759 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560811 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.560951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.560913 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.561036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.561004 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.561096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.561076 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.562665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.562643 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.562915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.562898 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.582191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.582156 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.583027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.583001 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97m9s\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\") pod \"e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:04.598611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.598577 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:04.699308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:04.696266 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:05.151899 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:33:05.151870 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bd5027_e611_438c_a0cb_c3b092154d30.slice/crio-1db02d3a2a651c853035a874e4aee5a6249d87bea83491a363a1c7ef7bc3a6e7 WatchSource:0}: Error finding container 1db02d3a2a651c853035a874e4aee5a6249d87bea83491a363a1c7ef7bc3a6e7: Status 404 returned error can't find the container with id 1db02d3a2a651c853035a874e4aee5a6249d87bea83491a363a1c7ef7bc3a6e7 Oct 30 09:33:05.153975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.153952 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:33:05.154767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.154733 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:05.154841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.154806 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:05.154841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.154836 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:05.280329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.280004 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:33:05.355154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.355111 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerStarted","Data":"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8"} Oct 30 09:33:05.355331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.355185 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerStarted","Data":"1db02d3a2a651c853035a874e4aee5a6249d87bea83491a363a1c7ef7bc3a6e7"} Oct 30 09:33:05.398863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.398748 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podStartSLOduration=1.398728386 podStartE2EDuration="1.398728386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:05.397988347 +0000 UTC m=+2349.121415517" watchObservedRunningTime="2026-10-30 09:33:05.398728386 +0000 UTC m=+2349.122155533" Oct 30 09:33:05.697748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:05.697673 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:06.586713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:06.586674 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:06.704297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:06.704266 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:07.370293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:07.370252 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:07.371146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:07.371125 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:33:07.729229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:07.729126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:07.827134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:07.827098 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:08.366171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:08.366139 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:09.420163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:09.420110 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:09.520600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:09.520564 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:10.313662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:10.313618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:10.412881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:10.412850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:10.525740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:10.525703 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:11.740288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:11.740252 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:12.162656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:12.162620 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:12.614929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:12.614895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:13.861416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:13.861381 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:15.455567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:15.455531 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:16.920902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:16.920870 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:18.253373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.253335 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:18.353829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.353787 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:18.586662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.586622 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:33:18.588246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.586904 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" containerID="cri-o://9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7" gracePeriod=30 Oct 30 09:33:18.611658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.611623 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:18.716082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:18.715875 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:19.225726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.225690 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq"] Oct 30 09:33:19.232674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.232644 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq"] Oct 30 09:33:19.232903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.232770 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.248403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.248368 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.248518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.248458 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.248566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.248519 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfj5\" (UniqueName: \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.251375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.251357 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:33:19.251471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.251432 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:33:19.349453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.349416 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfj5\" (UniqueName: \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.349902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.349513 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.349902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.349585 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.350272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.350253 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.351825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.351799 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.368392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.368367 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfj5\" (UniqueName: \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\") pod \"maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.555957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.555929 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:19.889804 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:33:19.889768 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c74908_385a_4de6_afaf_89b509951f86.slice/crio-489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb WatchSource:0}: Error finding container 489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb: Status 404 returned error can't find the container with id 489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb Oct 30 09:33:19.893509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:19.893483 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq"] Oct 30 09:33:20.396948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:20.396914 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:20.567247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:20.564903 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" event={"ID":"41c74908-385a-4de6-afaf-89b509951f86","Type":"ContainerStarted","Data":"a0f4e6aabe6b69426537241b57acdc2ab65763850fb08106faf21eced28c34ce"} Oct 30 09:33:20.567247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:20.564971 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" event={"ID":"41c74908-385a-4de6-afaf-89b509951f86","Type":"ContainerStarted","Data":"489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb"} Oct 30 09:33:20.629243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:20.627820 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" podStartSLOduration=1.627788096 podStartE2EDuration="1.627788096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:20.610647825 +0000 UTC m=+2364.334074969" watchObservedRunningTime="2026-10-30 09:33:20.627788096 +0000 UTC m=+2364.351215250" Oct 30 09:33:21.575484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:21.575447 4920 generic.go:356] "Generic (PLEG): container finished" podID="41c74908-385a-4de6-afaf-89b509951f86" containerID="a0f4e6aabe6b69426537241b57acdc2ab65763850fb08106faf21eced28c34ce" exitCode=0 Oct 30 09:33:21.575902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:21.575547 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" event={"ID":"41c74908-385a-4de6-afaf-89b509951f86","Type":"ContainerDied","Data":"a0f4e6aabe6b69426537241b57acdc2ab65763850fb08106faf21eced28c34ce"} Oct 30 09:33:22.104242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.100500 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:22.235987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.235907 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:22.866718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.866636 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:22.959582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.959537 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\") pod \"41c74908-385a-4de6-afaf-89b509951f86\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " Oct 30 09:33:22.959791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.959596 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\") pod \"41c74908-385a-4de6-afaf-89b509951f86\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " Oct 30 09:33:22.959791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.959669 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\" (UniqueName: \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\") pod \"41c74908-385a-4de6-afaf-89b509951f86\" (UID: \"41c74908-385a-4de6-afaf-89b509951f86\") " Oct 30 09:33:22.960318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.960283 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca" pod "41c74908-385a-4de6-afaf-89b509951f86" (UID: "41c74908-385a-4de6-afaf-89b509951f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:22.960527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.960501 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c74908-385a-4de6-afaf-89b509951f86-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:22.962631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.962602 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca" pod "41c74908-385a-4de6-afaf-89b509951f86" (UID: "41c74908-385a-4de6-afaf-89b509951f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:33:22.963040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:22.963016 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5" pod "41c74908-385a-4de6-afaf-89b509951f86" (UID: "41c74908-385a-4de6-afaf-89b509951f86"). InnerVolumeSpecName "kube-api-access-twfj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:23.074243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.073416 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c74908-385a-4de6-afaf-89b509951f86-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.074243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.073462 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfj5\" (UniqueName: \"kubernetes.io/projected/41c74908-385a-4de6-afaf-89b509951f86-kube-api-access-twfj5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.594719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.594680 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" Oct 30 09:33:23.594719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.594708 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq" event={"ID":"41c74908-385a-4de6-afaf-89b509951f86","Type":"ContainerDied","Data":"489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb"} Oct 30 09:33:23.594991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.594761 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489c106b3860e14f10f540cc48511afe7d35a02059cf22a532bc6d73c8b3a6eb" Oct 30 09:33:23.852573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.852477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:23.867576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.867549 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:33:23.988185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988148 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988375 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988472 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988511 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988546 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.988616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988569 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988611 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988656 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988732 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988765 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\") pod \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\" (UID: \"bb8adee7-a31f-4b9b-8bf7-e8604e434528\") " Oct 30 09:33:23.988843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.988799 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.989288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989262 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.989412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989393 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.989412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989395 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:23.989541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989413 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.989541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989429 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:23.989694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.989671 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.991981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.991937 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:33:23.992143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.992125 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "kube-api-access-8mqp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:23.993354 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.993318 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:33:23.993457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:23.993363 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token" pod "bb8adee7-a31f-4b9b-8bf7-e8604e434528" (UID: "bb8adee7-a31f-4b9b-8bf7-e8604e434528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:24.090841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090803 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.090841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090834 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqp5\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-kube-api-access-8mqp5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.090841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090845 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.091121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090854 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.091121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090862 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8adee7-a31f-4b9b-8bf7-e8604e434528-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.091121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.090871 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8adee7-a31f-4b9b-8bf7-e8604e434528-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:24.605190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.605152 4920 generic.go:356] "Generic (PLEG): container finished" podID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerID="9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7" exitCode=0 Oct 30 09:33:24.605507 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.605471 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerDied","Data":"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7"} Oct 30 09:33:24.605630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.605538 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" event={"ID":"bb8adee7-a31f-4b9b-8bf7-e8604e434528","Type":"ContainerDied","Data":"94d280f1591e6fe887e48fde04026abb95cea10516d4c50f07f30141ec93c72a"} Oct 30 09:33:24.605630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.605542 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf" Oct 30 09:33:24.605630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.605571 4920 scope.go:122] "RemoveContainer" containerID="9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7" Oct 30 09:33:24.616969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.616946 4920 scope.go:122] "RemoveContainer" containerID="2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700" Oct 30 09:33:24.629524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.629504 4920 scope.go:122] "RemoveContainer" containerID="9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7" Oct 30 09:33:24.629883 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:33:24.629860 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7\": container with ID starting with 9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7 not found: ID does not exist" containerID="9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7" Oct 30 09:33:24.629980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.629897 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7"} err="failed to get container status \"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7\": rpc error: code = NotFound desc = could not find container \"9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7\": container with ID starting with 9db67e318a5814bbed4c077fb8b94e62ba65d4abe0444f6f586f92a73b5905b7 not found: ID does not exist" Oct 30 09:33:24.629980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.629923 4920 scope.go:122] "RemoveContainer" containerID="2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700" Oct 30 09:33:24.630250 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:33:24.630232 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700\": container with ID starting with 2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700 not found: ID does not exist" containerID="2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700" Oct 30 09:33:24.630324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.630259 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700"} err="failed to get container status \"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700\": rpc error: code = NotFound desc = could not find container \"2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700\": container with ID starting with 2d9578b3f2caa327f07b74c60f398fcf9b6f926c2e9d66c91d52a907e5da4700 not found: ID does not exist" Oct 30 09:33:24.670964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.670877 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:33:24.683356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:24.683320 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d87c85-openshift-default-dfd7d8cc7-7rspf"] Oct 30 09:33:25.223481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:25.223444 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:25.355985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:25.355952 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:26.416155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:26.416119 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" path="/var/lib/kubelet/pods/bb8adee7-a31f-4b9b-8bf7-e8604e434528/volumes" Oct 30 09:33:26.887635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:26.887596 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:28.650959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.650922 4920 generic.go:356] "Generic (PLEG): container finished" podID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerID="2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587" exitCode=0 Oct 30 09:33:28.651416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.650996 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerDied","Data":"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587"} Oct 30 09:33:28.651476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.651460 4920 scope.go:122] "RemoveContainer" containerID="2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587" Oct 30 09:33:28.653187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.653157 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:28.653322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.653245 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:28.653322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.653280 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.744123 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745097 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745117 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745176 4920 cpu_manager.go:414] "containerMap: removing container" podUID="41c74908-385a-4de6-afaf-89b509951f86" containerName="revoke-keys" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745186 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c74908-385a-4de6-afaf-89b509951f86" containerName="revoke-keys" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745362 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745380 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:28.749243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.745397 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c74908-385a-4de6-afaf-89b509951f86" containerName="revoke-keys" Oct 30 09:33:28.750592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.750455 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.774493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.774452 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-0dc148-serving-cert\"" Oct 30 09:33:28.779442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.779408 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:33:28.829303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.828822 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:28.872204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.871813 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqswb\" (UniqueName: \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.872204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.871857 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.872204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.871983 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.872204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.872093 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.918141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.918098 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:28.985018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.979381 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.985018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.979557 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.985018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.979621 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqswb\" (UniqueName: \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.985018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.979664 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:28.985018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:28.980577 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:29.013991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.011914 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:29.033036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.032955 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqswb\" (UniqueName: \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:29.046965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.046875 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:29.047680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.047589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:29.130804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.130704 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:29.459963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.459939 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:33:29.665675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.665405 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerStarted","Data":"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f"} Oct 30 09:33:29.666914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:29.666887 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" event={"ID":"7ca2f98d-2b90-4d40-82bf-abb15f0f7105","Type":"ContainerStarted","Data":"c21aeb2e88b7a4b8e9ea14f8bfb3c1c0c85707239a677260013c83ac70f30186"} Oct 30 09:33:30.229407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.215540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:30.319242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.313751 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:33:30.319242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.313868 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 09:33:30.319242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.313912 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 09:33:30.553908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.553878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:30.690398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.689674 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" event={"ID":"7ca2f98d-2b90-4d40-82bf-abb15f0f7105","Type":"ContainerStarted","Data":"32a84569db7e9881c9b92dd6b2146841d8ac51a323d5bf94599cf0cf876fa1cc"} Oct 30 09:33:30.732458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:30.732275 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" podStartSLOduration=2.354504376 podStartE2EDuration="2.732254843s" totalImagesPullingTime="377.750467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:30.73088478 +0000 UTC m=+2374.454311926" watchObservedRunningTime="2026-10-30 09:33:30.732254843 +0000 UTC m=+2374.455681988" Oct 30 09:33:31.157092 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:33:31.157053 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:33:31.325834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:31.324635 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 09:33:31.325834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:31.324711 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 09:33:31.721242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:31.697202 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:32.177933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:32.177904 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:32.312596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:32.312557 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 09:33:32.313272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:32.312864 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 09:33:33.311667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:33.311627 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 09:33:33.312300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:33.312264 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 09:33:33.762393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:33.762287 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:34.326369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:34.326332 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 09:33:34.327001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:34.326967 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 09:33:34.583172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:34.583095 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:33:34.865724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:34.865598 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:35.328373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:35.328339 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:33:35.330967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:35.330038 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:33:35.331320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:35.331288 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:33:35.518144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:35.518112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:36.312256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:36.312188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:36.778338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:36.778252 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:37.231568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.231527 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:33:37.232531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.232502 4920 cpu_manager.go:414] "containerMap: removing container" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:37.232531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.232527 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8adee7-a31f-4b9b-8bf7-e8604e434528" containerName="istio-proxy" Oct 30 09:33:37.237560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.237534 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.249126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.249085 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:33:37.251551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.251522 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:37.256874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.256853 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7a213d-openshift-default-dockercfg-txlvp\"" Oct 30 09:33:37.339510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.339473 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.339970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.339941 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjlrv\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.340114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.340097 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.340276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.340261 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.340511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.340492 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.340638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.340622 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.345514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.345464 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.345789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.345757 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.346039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.346015 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.382799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.382664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:37.464240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.464185 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjlrv\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.466199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.466174 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.466720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.466700 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.467257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.467143 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.467505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.467487 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.467916 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.467900 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.486280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.486190 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.486505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.486485 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.486630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.486617 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.486872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.486852 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.487485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.487460 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.487871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.487851 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.488670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.488647 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.489050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.489029 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.498657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.498583 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.536255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.510848 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.536255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.516072 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:37.536255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.521295 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjlrv\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.552253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.552120 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\") pod \"e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.576635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.563473 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:37.736070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:37.736037 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:33:38.038772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.037151 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:33:38.041002 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:33:38.040977 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5716962_c02e_4388_83b0_b11c9023bc90.slice/crio-847ea1c61be36984ed68642e8b6f869fe8d831fb23acb2c6f983e4b885bd57af WatchSource:0}: Error finding container 847ea1c61be36984ed68642e8b6f869fe8d831fb23acb2c6f983e4b885bd57af: Status 404 returned error can't find the container with id 847ea1c61be36984ed68642e8b6f869fe8d831fb23acb2c6f983e4b885bd57af Oct 30 09:33:38.044509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.044413 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:38.044591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.044547 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:38.044591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.044575 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:38.956110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.956075 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:38.974950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.974913 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerStarted","Data":"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b"} Oct 30 09:33:38.975152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:38.974964 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerStarted","Data":"847ea1c61be36984ed68642e8b6f869fe8d831fb23acb2c6f983e4b885bd57af"} Oct 30 09:33:39.063286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:39.060768 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podStartSLOduration=2.060741649 podStartE2EDuration="2.060741649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:39.037739736 +0000 UTC m=+2382.761166896" watchObservedRunningTime="2026-10-30 09:33:39.060741649 +0000 UTC m=+2382.784168797" Oct 30 09:33:39.565997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:39.565961 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:39.572005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:39.571974 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:39.983511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:39.983431 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:39.984491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:39.984469 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:33:40.827313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:40.823728 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:41.391946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:41.391903 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:41.492271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:41.492227 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:41.593752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:41.593715 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:42.600929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:42.600881 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:43.130531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:43.130471 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:43.230786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:43.230739 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:44.450038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:44.450000 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:46.057611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.057547 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:46.826964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.826863 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:33:46.833693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.833663 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.859678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.859645 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:33:46.921860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.921828 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.921909 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.921956 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922072 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922112 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922140 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922204 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9v8p\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922336 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:46.922433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:46.922382 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.023652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023611 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.023875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023709 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.023875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023751 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.023875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023811 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.023875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023855 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023893 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.023920 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9v8p\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.024023 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.024134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.024069 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.024535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.024511 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.025091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.024844 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.025091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.024947 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.025091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.025000 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.025091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.025044 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.027202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.027172 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.030766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.030741 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.044306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.044273 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9v8p\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.044608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.044585 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.046168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.046144 4920 generic.go:356] "Generic (PLEG): container finished" podID="2947718f-2e49-48c9-abf1-0a224f149c61" containerID="d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29" exitCode=0 Oct 30 09:33:47.046297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.046194 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerDied","Data":"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29"} Oct 30 09:33:47.046676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.046660 4920 scope.go:122] "RemoveContainer" containerID="d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29" Oct 30 09:33:47.048552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.048523 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.048636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.048596 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.048686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.048638 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.169867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.169837 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:47.634565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.634534 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:33:47.637629 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:33:47.636576 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7795d4_23df_4ff0_8d58_89858fec7b26.slice/crio-adeeba2ef774d1ff511834f4ee02552dc0bd14607c2793638d286d89490bb002 WatchSource:0}: Error finding container adeeba2ef774d1ff511834f4ee02552dc0bd14607c2793638d286d89490bb002: Status 404 returned error can't find the container with id adeeba2ef774d1ff511834f4ee02552dc0bd14607c2793638d286d89490bb002 Oct 30 09:33:47.640135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.640103 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.640280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.640174 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.640280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.640205 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.747936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:47.747906 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:48.059573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.059530 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" event={"ID":"5b7795d4-23df-4ff0-8d58-89858fec7b26","Type":"ContainerStarted","Data":"0a697d3e0334a005fc15a1ee082ab115e805fc69f1ec93e3a7a525b9e256c71c"} Oct 30 09:33:48.059869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.059850 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" event={"ID":"5b7795d4-23df-4ff0-8d58-89858fec7b26","Type":"ContainerStarted","Data":"adeeba2ef774d1ff511834f4ee02552dc0bd14607c2793638d286d89490bb002"} Oct 30 09:33:48.061972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.061952 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerStarted","Data":"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af"} Oct 30 09:33:48.170533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.170401 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:48.172343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.172312 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:48.172574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.172375 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:48.172648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.172532 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podStartSLOduration=2.172517863 podStartE2EDuration="2.172517863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:48.127615942 +0000 UTC m=+2391.851043086" watchObservedRunningTime="2026-10-30 09:33:48.172517863 +0000 UTC m=+2391.895945006" Oct 30 09:33:48.930242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.929598 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:33:48.934238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.930983 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:33:48.934238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:48.931035 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:33:49.126376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.126336 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:33:49.170487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.170458 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:49.170725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.170691 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:49.631241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.626056 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:49.763306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.762061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:49.950790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.950706 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:33:49.951366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:49.951263 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:33:50.183598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:50.183561 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:50.184192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:50.184158 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:50.931242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:50.930592 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:33:50.931242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:50.930658 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:33:50.984265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:50.984120 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:51.176462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.176424 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:51.177114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.177079 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:51.177308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.176995 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:33:51.177535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.177509 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" podUID="5782e737-46bd-462d-8d79-47895448870c" containerName="maas-api" containerID="cri-o://2c1aaec4b620ff011ccce437e53c0e149661a7a7915582cff2f296ad26fc5fca" gracePeriod=30 Oct 30 09:33:51.179229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.179175 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:51.275878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.275846 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:51.381100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.380978 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:51.481562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.481451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:51.942242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.942050 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:33:51.942242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:51.942110 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:33:52.067242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.009619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:52.096452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.096401 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:52.233296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.229379 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:52.233296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.229463 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:52.265617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.265572 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.268678 4920 generic.go:356] "Generic (PLEG): container finished" podID="5782e737-46bd-462d-8d79-47895448870c" containerID="2c1aaec4b620ff011ccce437e53c0e149661a7a7915582cff2f296ad26fc5fca" exitCode=0 Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.268744 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" event={"ID":"5782e737-46bd-462d-8d79-47895448870c","Type":"ContainerDied","Data":"2c1aaec4b620ff011ccce437e53c0e149661a7a7915582cff2f296ad26fc5fca"} Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.268842 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" event={"ID":"5782e737-46bd-462d-8d79-47895448870c","Type":"ContainerDied","Data":"f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0"} Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.268869 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3bd695aa3910f86c5b789fb16b177724a27410e6b4ea9f5d313eace919dfbd0" Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.269406 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:33:52.270239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.269850 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq"] Oct 30 09:33:52.273948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.273924 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-15e1cc-bpvfq"] Oct 30 09:33:52.446380 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.425930 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\" (UniqueName: \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\") pod \"5782e737-46bd-462d-8d79-47895448870c\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " Oct 30 09:33:52.481275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.447045 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\") pod \"5782e737-46bd-462d-8d79-47895448870c\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " Oct 30 09:33:52.481275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.480385 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\") pod \"5782e737-46bd-462d-8d79-47895448870c\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " Oct 30 09:33:52.481275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.480483 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\") pod \"5782e737-46bd-462d-8d79-47895448870c\" (UID: \"5782e737-46bd-462d-8d79-47895448870c\") " Oct 30 09:33:52.481589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.480056 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78" pod "5782e737-46bd-462d-8d79-47895448870c" (UID: "5782e737-46bd-462d-8d79-47895448870c"). InnerVolumeSpecName "kube-api-access-c8j78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:52.481825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.481798 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca" pod "5782e737-46bd-462d-8d79-47895448870c" (UID: "5782e737-46bd-462d-8d79-47895448870c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:52.484581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.484507 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs" pod "5782e737-46bd-462d-8d79-47895448870c" (UID: "5782e737-46bd-462d-8d79-47895448870c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:33:52.484705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.484593 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls" pod "5782e737-46bd-462d-8d79-47895448870c" (UID: "5782e737-46bd-462d-8d79-47895448870c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:33:52.496153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.495396 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c74908-385a-4de6-afaf-89b509951f86" path="/var/lib/kubelet/pods/41c74908-385a-4de6-afaf-89b509951f86/volumes" Oct 30 09:33:52.536953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.536813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:52.583036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.583001 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8j78\" (UniqueName: \"kubernetes.io/projected/5782e737-46bd-462d-8d79-47895448870c-kube-api-access-c8j78\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.583036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.583033 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.583036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.583044 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5782e737-46bd-462d-8d79-47895448870c-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.583276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.583052 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5782e737-46bd-462d-8d79-47895448870c-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.639744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.639704 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:52.938243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.929001 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:33:52.938243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:52.929076 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:33:53.170234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.170180 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:53.170444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.170277 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:53.278374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.278082 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2" Oct 30 09:33:53.347241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.346851 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:33:53.357265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.356980 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-15e1cc-5874575484-zg9s2"] Oct 30 09:33:53.773394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.773353 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:53.976237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.967516 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:33:53.989238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:53.987652 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:33:54.176500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:54.176346 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 09:33:54.176500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:54.176434 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 09:33:54.315908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:54.315876 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:54.440024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:54.439941 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5782e737-46bd-462d-8d79-47895448870c" path="/var/lib/kubelet/pods/5782e737-46bd-462d-8d79-47895448870c/volumes" Oct 30 09:33:55.175595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:55.175567 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:55.176013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:55.175992 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:55.176673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:55.176656 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:33:56.046514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:56.046439 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:57.570683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:57.570644 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:58.298007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.297973 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:58.379940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.379841 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:58.493868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.493827 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:58.621680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.621582 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:33:58.622584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.622539 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" containerID="cri-o://5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f" gracePeriod=30 Oct 30 09:33:58.642379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:58.642353 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:59.236866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:59.236830 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:33:59.699620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:33:59.699584 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:00.142895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.142845 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:34:00.143869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.143843 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5782e737-46bd-462d-8d79-47895448870c" containerName="maas-api" Oct 30 09:34:00.144042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.144020 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5782e737-46bd-462d-8d79-47895448870c" containerName="maas-api" Oct 30 09:34:00.144262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.144245 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5782e737-46bd-462d-8d79-47895448870c" containerName="maas-api" Oct 30 09:34:00.150955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.150931 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.220336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.220281 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-1413c8-openshift-default-dockercfg-8sr9v\"" Oct 30 09:34:00.222823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.222789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:00.224112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.224079 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:34:00.230371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230341 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230409 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230539 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230667 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230737 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230854 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.230926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230895 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.231009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.230982 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.231130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.231115 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf84t\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345181 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345355 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf84t\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345420 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345464 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345489 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345818 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.354144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.347417 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.355068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.355008 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.393622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.345581 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.394271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.394247 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.395006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.394983 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.406275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.400350 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.406275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.400601 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.453338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.425823 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.453338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.427726 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.453338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.427757 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.494899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.463699 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:00.495127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.495092 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf84t\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.496265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.495421 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\") pod \"e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.532180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.529784 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:00.894084 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:34:00.894052 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5fb028_3125_4c93_98ec_c890f82e3d85.slice/crio-6a88abd9c7f9909bcc9ab896665e15f1f0b95bf91b5afd974ed1addc525ea63d WatchSource:0}: Error finding container 6a88abd9c7f9909bcc9ab896665e15f1f0b95bf91b5afd974ed1addc525ea63d: Status 404 returned error can't find the container with id 6a88abd9c7f9909bcc9ab896665e15f1f0b95bf91b5afd974ed1addc525ea63d Oct 30 09:34:00.896077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.896048 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:34:00.896946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.896916 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:00.897009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.896995 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:00.897050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:00.897028 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:01.050259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.050188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:01.204040 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:34:01.203940 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:34:01.520424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.520334 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerStarted","Data":"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888"} Oct 30 09:34:01.520424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.520390 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerStarted","Data":"6a88abd9c7f9909bcc9ab896665e15f1f0b95bf91b5afd974ed1addc525ea63d"} Oct 30 09:34:01.533983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.533950 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:01.540195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.540165 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:01.572711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:01.572659 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podStartSLOduration=1.5726457809999999 podStartE2EDuration="1.572645781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:01.571500304 +0000 UTC m=+2405.294927447" watchObservedRunningTime="2026-10-30 09:34:01.572645781 +0000 UTC m=+2405.296072924" Oct 30 09:34:02.529597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:02.529565 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:02.531076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:02.531051 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:34:02.645233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:02.645192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:02.741369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:02.740304 4920 patch_prober.go:29] interesting pod/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:34:02.741369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:02.740365 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:34:03.252021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:03.251995 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:34:03.344791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:03.344766 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:34:03.721241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:03.721164 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:34:04.017571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.017535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:04.027708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.027684 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:34:04.133194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.133156 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:04.197810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.197727 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.197810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.197797 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.197852 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.197886 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.197925 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198064 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198105 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198139 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198174 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:04.198290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198285 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\") pod \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\" (UID: \"6acf74bf-7ee2-474d-af8e-b0d857263c6b\") " Oct 30 09:34:04.198622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198287 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:04.198622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198352 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:04.198755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198634 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:34:04.199059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.198998 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.199059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.199031 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.199059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.199045 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.199059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.199061 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.199434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.199075 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.200708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.200684 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:04.200981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.200952 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:34:04.201078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.200974 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "kube-api-access-2x67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:04.201078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.200993 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy" pod "6acf74bf-7ee2-474d-af8e-b0d857263c6b" (UID: "6acf74bf-7ee2-474d-af8e-b0d857263c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:04.300306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.300275 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x67s\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-kube-api-access-2x67s\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.300306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.300302 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.300306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.300312 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.300567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.300320 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6acf74bf-7ee2-474d-af8e-b0d857263c6b-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:34:04.416895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.416866 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:34:04.417529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.417500 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:34:04.417529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.417520 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:34:04.417727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.417557 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:34:04.417727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.417565 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:34:04.417838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.417727 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:34:04.421664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.421645 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.437341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.437302 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7a213d-serving-cert\"" Oct 30 09:34:04.448110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.444434 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:34:04.501028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.500976 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:04.503309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.503276 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.503497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.503344 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9sz\" (UniqueName: \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.503497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.503392 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.503497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.503445 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.549080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.549043 4920 generic.go:356] "Generic (PLEG): container finished" podID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerID="5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f" exitCode=0 Oct 30 09:34:04.549320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.549097 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" Oct 30 09:34:04.549320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.549127 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerDied","Data":"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f"} Oct 30 09:34:04.549320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.549192 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm" event={"ID":"6acf74bf-7ee2-474d-af8e-b0d857263c6b","Type":"ContainerDied","Data":"96ae90ff5b86f8e760cae3609747c88fcc383e74eaf6e8db073daade6584f77a"} Oct 30 09:34:04.549320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.549237 4920 scope.go:122] "RemoveContainer" containerID="5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f" Oct 30 09:34:04.562949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.562918 4920 scope.go:122] "RemoveContainer" containerID="77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b" Oct 30 09:34:04.573234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.573197 4920 scope.go:122] "RemoveContainer" containerID="5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f" Oct 30 09:34:04.573539 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:34:04.573514 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f\": container with ID starting with 5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f not found: ID does not exist" containerID="5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f" Oct 30 09:34:04.573618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.573547 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f"} err="failed to get container status \"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f\": rpc error: code = NotFound desc = could not find container \"5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f\": container with ID starting with 5ac0238926dfc86d6a1a01d380b42dfad7b59e10e5987ef15435a6ba41036c7f not found: ID does not exist" Oct 30 09:34:04.573618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.573565 4920 scope.go:122] "RemoveContainer" containerID="77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b" Oct 30 09:34:04.573869 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:34:04.573849 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b\": container with ID starting with 77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b not found: ID does not exist" containerID="77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b" Oct 30 09:34:04.573941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.573877 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b"} err="failed to get container status \"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b\": rpc error: code = NotFound desc = could not find container \"77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b\": container with ID starting with 77ba554e4ce0b7c2a2dd55e7ef67dc7cd6200d85a25768bc530b4e7ba2a47c4b not found: ID does not exist" Oct 30 09:34:04.603420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.603381 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:04.604362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.604336 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:34:04.605464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.605430 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.605597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.605492 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9sz\" (UniqueName: \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.605597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.605537 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.605597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.605587 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.607125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.606943 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.609054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.609030 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.609623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.609580 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-15e1cc-openshift-default-6f756dc5f7-6zfzm"] Oct 30 09:34:04.610453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.610430 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.644269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.642549 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9sz\" (UniqueName: \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\") pod \"maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:04.754112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:04.754030 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:05.273620 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:34:05.273563 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6595eb_f73c_4c26_89c9_3a693410528f.slice/crio-8f51dc1cf7e1822d126645d4bceabe1c3759fa17603d9b429abd826ed34ccd19 WatchSource:0}: Error finding container 8f51dc1cf7e1822d126645d4bceabe1c3759fa17603d9b429abd826ed34ccd19: Status 404 returned error can't find the container with id 8f51dc1cf7e1822d126645d4bceabe1c3759fa17603d9b429abd826ed34ccd19 Oct 30 09:34:05.277922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:05.277891 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:34:05.559730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:05.559682 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" event={"ID":"1b6595eb-f73c-4c26-89c9-3a693410528f","Type":"ContainerStarted","Data":"8f51dc1cf7e1822d126645d4bceabe1c3759fa17603d9b429abd826ed34ccd19"} Oct 30 09:34:05.811083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:05.810208 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:05.896865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:05.896828 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:06.407901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:06.407863 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" path="/var/lib/kubelet/pods/6acf74bf-7ee2-474d-af8e-b0d857263c6b/volumes" Oct 30 09:34:06.570631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:06.570594 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" event={"ID":"1b6595eb-f73c-4c26-89c9-3a693410528f","Type":"ContainerStarted","Data":"c787e7d5460c53ceb939139fb4089c67b3ece26aaede7effa87793efd8e03107"} Oct 30 09:34:06.570910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:06.570673 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:06.579914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:06.579886 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:06.614167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:06.614119 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" podStartSLOduration=2.276468123 podStartE2EDuration="2.614106113s" totalImagesPullingTime="337.63799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:06.611999573 +0000 UTC m=+2410.335426720" watchObservedRunningTime="2026-10-30 09:34:06.614106113 +0000 UTC m=+2410.337533255" Oct 30 09:34:08.119094 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:08.119039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:09.643429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:09.643393 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:11.195236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:11.195183 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:12.117982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:12.117947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:12.219073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:12.219016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:12.591183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:12.591154 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:34:12.702230 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:12.702185 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:13.404970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:13.404906 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:14.375356 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:14.375315 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:15.952711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:15.952678 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:16.613044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:16.613003 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:17.560833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:17.560795 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:19.072032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:19.071997 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:20.534018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:20.533980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:22.143363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:22.143327 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:23.707437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:23.707402 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:25.165146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:25.165108 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:26.727112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:26.727074 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:26.827234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:26.827191 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:26.940544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:26.940507 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:28.444049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:28.444014 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:28.809239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:28.805413 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:34:29.392359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:29.392318 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:29.493237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:29.493025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:30.168154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:30.168121 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:31.153972 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:34:31.153931 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:34:31.909679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:31.909642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:32.331127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:32.331034 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:32.441966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:32.441931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:33.580517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:33.580481 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:35.179310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:35.179277 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:36.736160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:36.736090 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:38.394696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:38.394659 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:39.985190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:39.985152 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:39.985662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:39.985210 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:40.075639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.075604 4920 generic.go:356] "Generic (PLEG): container finished" podID="d5716962-c02e-4388-83b0-b11c9023bc90" containerID="297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b" exitCode=0 Oct 30 09:34:40.075825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.075674 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerDied","Data":"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b"} Oct 30 09:34:40.076142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.076116 4920 scope.go:122] "RemoveContainer" containerID="297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b" Oct 30 09:34:40.077754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.077722 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:40.077841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.077794 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:40.077841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:40.077828 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:41.085327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.085287 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerStarted","Data":"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae"} Oct 30 09:34:41.112757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.112722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:41.565912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.565873 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:34:41.566130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.566108 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:41.566186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.566167 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:41.662695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:41.662661 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:42.565677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:42.565642 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:42.566113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:42.565693 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:43.137572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:43.137535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:43.565648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:43.565613 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:43.565834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:43.565673 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:44.565956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:44.565916 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:44.566570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:44.566145 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:44.615269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:44.614664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:45.566528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:45.566493 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 09:34:45.566978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:45.566549 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 09:34:46.089533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:46.089493 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:46.614110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:46.614065 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:34:46.615079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:46.615052 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:34:46.617954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:46.617933 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:34:47.705429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:47.705390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:49.487091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:49.487058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:51.074045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:51.074007 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:51.185855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:51.185801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:51.274616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:51.274581 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:52.784790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:52.784758 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:54.136469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:54.136427 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:54.494776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:54.494677 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:54.996450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:54.996412 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:56.116441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:56.116406 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:57.436875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:57.436843 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:57.767327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:57.767244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:34:59.318996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.318964 4920 generic.go:356] "Generic (PLEG): container finished" podID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerID="498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888" exitCode=0 Oct 30 09:34:59.319589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.319040 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerDied","Data":"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888"} Oct 30 09:34:59.319673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.319646 4920 scope.go:122] "RemoveContainer" containerID="498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888" Oct 30 09:34:59.321567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.321539 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:59.321662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.321630 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:59.321708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.321687 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:59.368922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:34:59.368886 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:00.346054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:00.345203 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerStarted","Data":"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08"} Oct 30 09:35:00.534464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:00.534424 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:00.536190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:00.536160 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:00.536351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:00.536245 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:00.636068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:00.635980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:01.077151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:01.077113 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:01.237425 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:35:01.237384 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:35:01.534948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:01.534908 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:01.535413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:01.535004 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:02.522729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:02.522677 4920 scope.go:122] "RemoveContainer" containerID="112a7298aa7fcb0bad317ca5a44005d60b62ad2f025b067dc39eba98a68fccf3" Oct 30 09:35:02.531240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:02.529852 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:02.534867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:02.534841 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:02.535024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:02.534895 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:02.662687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:02.662655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:03.033361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:03.033321 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp"] Oct 30 09:35:03.042029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:03.041996 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bc34e8-qwtkp"] Oct 30 09:35:03.534191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:03.534158 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:03.534399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:03.534255 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:04.250508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:04.250472 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:04.404384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:04.404344 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd15c3a-112f-4868-9ed2-c8749e162237" path="/var/lib/kubelet/pods/bdd15c3a-112f-4868-9ed2-c8749e162237/volumes" Oct 30 09:35:04.534152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:04.534125 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:04.534364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:04.534186 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:05.534700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:05.534657 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 09:35:05.535266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:05.534743 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 09:35:05.787178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:05.787092 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:06.539955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:06.539918 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:06.540976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:06.540953 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:07.032249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:07.032123 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx"] Oct 30 09:35:07.038472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:07.038439 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39b03bff-5kmpx"] Oct 30 09:35:07.353561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:07.353526 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:08.399241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:08.399130 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8f1221-674b-4993-8b67-cfe5ea6d9177" path="/var/lib/kubelet/pods/aa8f1221-674b-4993-8b67-cfe5ea6d9177/volumes" Oct 30 09:35:08.852701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:08.852660 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:08.965800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:08.965757 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:35:10.682777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:10.682738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:12.317243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:12.316638 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:13.036589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:13.036544 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n"] Oct 30 09:35:13.046312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:13.046281 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-552eca70-mjs9n"] Oct 30 09:35:13.867181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:13.867139 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:14.402242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:14.399259 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f22005-cec9-45c0-a3a8-674d16e0bce3" path="/var/lib/kubelet/pods/f8f22005-cec9-45c0-a3a8-674d16e0bce3/volumes" Oct 30 09:35:15.354042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:15.354011 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:16.861113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:16.861077 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:18.432465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:18.432423 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:20.058413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.058373 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:20.627305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.627256 4920 generic.go:356] "Generic (PLEG): container finished" podID="36bd5027-e611-438c-a0cb-c3b092154d30" containerID="9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8" exitCode=0 Oct 30 09:35:20.627305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.627291 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerDied","Data":"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8"} Oct 30 09:35:20.627841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.627822 4920 scope.go:122] "RemoveContainer" containerID="9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8" Oct 30 09:35:20.629788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.629754 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:20.629892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.629841 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:20.629956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:20.629889 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:21.641920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:21.641886 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerStarted","Data":"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec"} Oct 30 09:35:21.663612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:21.663578 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:21.710243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:21.706024 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:35:21.710243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:21.706168 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:21.710243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:21.706251 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:22.370800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:22.370751 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:35:22.722475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:22.722369 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:22.722475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:22.722439 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:22.980695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:22.980608 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:23.236341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:23.231204 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:23.714383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:23.714328 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:23.714730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:23.714680 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:24.704616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:24.704577 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:24.705284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:24.705250 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:24.840414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:24.840383 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:25.718378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:25.718336 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:25.719045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:25.719009 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:26.525121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:26.525084 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:26.703625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:26.703585 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 09:35:26.703927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:26.703895 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 09:35:27.703414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:27.703380 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:35:27.704569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:27.704543 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:35:28.182642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:28.182607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:29.770828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:29.770789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:31.136568 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:35:31.136531 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:35:31.489892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:31.489800 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:33.038460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:33.038426 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:33.933115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:33.933078 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:34.028453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:34.028413 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:34.814365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:34.814329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:36.471640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:36.471604 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:38.006086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:38.006053 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:39.433953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:39.433915 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:40.039398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:40.039361 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:40.157650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:40.157612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:40.408725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:40.408616 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:40.744074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:40.743984 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:42.034759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:42.034707 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:43.537626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:43.537589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:44.486351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:44.486310 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:45.045632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:45.045595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:45.320556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:45.320450 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:46.818362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:46.818325 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:47.118080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:47.117984 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:47.253234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:47.253186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:47.448612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:47.448523 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:35:47.449087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:47.449053 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" containerID="cri-o://4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08" gracePeriod=30 Oct 30 09:35:47.456106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:47.456063 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:48.123970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.123926 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:48.457782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.457711 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:48.569048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.569009 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:35:48.570285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.570247 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6acf74bf-7ee2-474d-af8e-b0d857263c6b" containerName="istio-proxy" Oct 30 09:35:48.574896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.574871 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.600012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.599979 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:35:48.606648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.606614 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d3047c-openshift-default-dockercfg-djf2k\"" Oct 30 09:35:48.625661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.625629 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:48.723297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.700312 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.724868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.724837 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.725090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725069 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.725292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725273 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.725449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725433 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.725601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725583 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.725746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725730 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.729245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.725902 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.729485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.729463 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvjn\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831387 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831457 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831511 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831551 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831615 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831670 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvjn\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831776 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831898 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.831924 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.833095 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.844240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.836037 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.851237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.846527 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.851237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.847068 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.851237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.847429 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:48.851237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.848168 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.901462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.901388 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.902002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.901911 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.916847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.904815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:48.918502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:48.917448 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvjn\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\") pod \"e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:49.193540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:49.193505 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:49.518186 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:35:49.518146 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f872428_e065_46fd_8593_f2d54392a0e8.slice/crio-9948f868f7d3efe324f0469791c9f9f58ae0ec2cb3eaf87d1c15b6dbcebdfdae WatchSource:0}: Error finding container 9948f868f7d3efe324f0469791c9f9f58ae0ec2cb3eaf87d1c15b6dbcebdfdae: Status 404 returned error can't find the container with id 9948f868f7d3efe324f0469791c9f9f58ae0ec2cb3eaf87d1c15b6dbcebdfdae Oct 30 09:35:49.519309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:49.519284 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:35:49.521099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:49.521071 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:49.521175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:49.521130 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:49.521175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:49.521165 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:35:50.005403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.005363 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" event={"ID":"0f872428-e065-46fd-8593-f2d54392a0e8","Type":"ContainerStarted","Data":"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327"} Oct 30 09:35:50.005594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.005451 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" event={"ID":"0f872428-e065-46fd-8593-f2d54392a0e8","Type":"ContainerStarted","Data":"9948f868f7d3efe324f0469791c9f9f58ae0ec2cb3eaf87d1c15b6dbcebdfdae"} Oct 30 09:35:50.050849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.050793 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" podStartSLOduration=2.050780454 podStartE2EDuration="2.050780454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:35:50.046803341 +0000 UTC m=+2513.770230504" watchObservedRunningTime="2026-10-30 09:35:50.050780454 +0000 UTC m=+2513.774207631" Oct 30 09:35:50.135416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.135382 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:50.204240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.194855 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:50.210569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.205231 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:50.450853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.450813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:50.601181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.601141 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:50.812734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:50.812347 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:51.014730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:51.014687 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:51.015811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:51.015789 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:35:51.248155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:51.248063 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:51.543929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:51.543890 4920 patch_prober.go:29] interesting pod/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:35:51.544103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:51.543955 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:35:52.063596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.063563 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:52.763208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.763183 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:52.791315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791282 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791363 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791391 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791410 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791532 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791587 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791610 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:52.791831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791790 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791814 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:52.791946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791834 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.791946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791818 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:35:52.791946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.791874 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\") pod \"1f5fb028-3125-4c93-98ec-c890f82e3d85\" (UID: \"1f5fb028-3125-4c93-98ec-c890f82e3d85\") " Oct 30 09:35:52.792114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792012 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:52.792114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792084 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:52.792556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792532 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.792633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792562 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.792633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792579 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.792633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792595 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.792633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.792609 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5fb028-3125-4c93-98ec-c890f82e3d85-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.793671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.793649 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:35:52.793889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.793875 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "kube-api-access-tf84t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:52.794151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.794136 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:52.794206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.794157 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy" pod "1f5fb028-3125-4c93-98ec-c890f82e3d85" (UID: "1f5fb028-3125-4c93-98ec-c890f82e3d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:52.893991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.893955 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.893991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.893987 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf84t\" (UniqueName: \"kubernetes.io/projected/1f5fb028-3125-4c93-98ec-c890f82e3d85-kube-api-access-tf84t\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.893991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.893999 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:52.894268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:52.894008 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5fb028-3125-4c93-98ec-c890f82e3d85-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:35:53.035559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.035526 4920 generic.go:356] "Generic (PLEG): container finished" podID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerID="4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08" exitCode=0 Oct 30 09:35:53.035750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.035595 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" Oct 30 09:35:53.035750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.035614 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerDied","Data":"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08"} Oct 30 09:35:53.035750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.035669 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz" event={"ID":"1f5fb028-3125-4c93-98ec-c890f82e3d85","Type":"ContainerDied","Data":"6a88abd9c7f9909bcc9ab896665e15f1f0b95bf91b5afd974ed1addc525ea63d"} Oct 30 09:35:53.035750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.035696 4920 scope.go:122] "RemoveContainer" containerID="4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08" Oct 30 09:35:53.048621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.048598 4920 scope.go:122] "RemoveContainer" containerID="498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888" Oct 30 09:35:53.058854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.058831 4920 scope.go:122] "RemoveContainer" containerID="4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08" Oct 30 09:35:53.059175 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:35:53.059155 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08\": container with ID starting with 4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08 not found: ID does not exist" containerID="4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08" Oct 30 09:35:53.060152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.059193 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08"} err="failed to get container status \"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08\": rpc error: code = NotFound desc = could not find container \"4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08\": container with ID starting with 4c435cea42c2587176436d2d290b6f955cb80208b96837686699b96a2d087c08 not found: ID does not exist" Oct 30 09:35:53.060152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.059249 4920 scope.go:122] "RemoveContainer" containerID="498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888" Oct 30 09:35:53.060152 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:35:53.059495 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888\": container with ID starting with 498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888 not found: ID does not exist" containerID="498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888" Oct 30 09:35:53.060152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.059519 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888"} err="failed to get container status \"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888\": rpc error: code = NotFound desc = could not find container \"498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888\": container with ID starting with 498e425ffb94cc31c9f576bcccd6c8b44f1f600bd8f632f417c2cb470a93b888 not found: ID does not exist" Oct 30 09:35:53.088750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.088716 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:35:53.102852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.102820 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-1413c8-openshift-default-5b8788488f-9thgz"] Oct 30 09:35:53.547343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.547302 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:53.917039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:53.916945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:54.402550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:54.400670 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" path="/var/lib/kubelet/pods/1f5fb028-3125-4c93-98ec-c890f82e3d85/volumes" Oct 30 09:35:55.195782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:55.195746 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:56.793949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:56.793876 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:58.364780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:58.364738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:35:59.834335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:59.834283 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:35:59.936953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:35:59.936913 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:01.073312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:01.073267 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw"] Oct 30 09:36:01.092284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:01.092178 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677fce03-fp2rw"] Oct 30 09:36:01.141265 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:36:01.141226 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:36:01.674419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:01.674383 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:02.398818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:02.398785 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec82776-9277-4509-84a5-2138e307f5a7" path="/var/lib/kubelet/pods/9ec82776-9277-4509-84a5-2138e307f5a7/volumes" Oct 30 09:36:02.564190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:02.564156 4920 scope.go:122] "RemoveContainer" containerID="5e26258d7913e3714bc2e763e319614d0aad3250bb41dfec3baea4de428f63c6" Oct 30 09:36:02.575531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:02.575502 4920 scope.go:122] "RemoveContainer" containerID="4c10f58cd89f47146b23c79e2b02de4768b55632e309d8ee5e55a2957dd0b0c8" Oct 30 09:36:02.590427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:02.590403 4920 scope.go:122] "RemoveContainer" containerID="7299f881d14e778c98c896809140e9bcbb3013dddfe8903d6a2cad0558f7e705" Oct 30 09:36:02.600068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:02.600045 4920 scope.go:122] "RemoveContainer" containerID="2c1aaec4b620ff011ccce437e53c0e149661a7a7915582cff2f296ad26fc5fca" Oct 30 09:36:03.367452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:03.367415 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:05.061533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:05.061498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:06.562178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:06.562144 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:08.191085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:08.191050 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:09.870567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:09.870537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:11.467120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:11.467083 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:13.180966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:13.180926 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:14.671560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:14.671523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:15.188181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.188126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:15.405202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.405135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:15.408483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.408454 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:36:15.408872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.408849 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" podUID="0f872428-e065-46fd-8593-f2d54392a0e8" containerName="istio-proxy" containerID="cri-o://53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327" gracePeriod=30 Oct 30 09:36:15.664743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.664671 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:15.828968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:15.828932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:16.326021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:16.325977 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:17.795432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:17.795392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:19.374911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:19.374869 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:20.790612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.790583 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:36:20.847452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847416 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.847452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847484 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:20.847805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847541 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.847805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847670 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.847805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847704 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.847805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847761 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.848033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847810 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.848033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.847904 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.848033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848014 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.848198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848051 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\") pod \"0f872428-e065-46fd-8593-f2d54392a0e8\" (UID: \"0f872428-e065-46fd-8593-f2d54392a0e8\") " Oct 30 09:36:20.849021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848497 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:20.849021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848643 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.849021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848668 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:20.849021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848743 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:20.849021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.848895 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:36:20.849378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.849308 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:20.850663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.850626 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:20.851115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.851093 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "kube-api-access-dqvjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:36:20.851265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.851202 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:36:20.851584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.851567 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo" pod "0f872428-e065-46fd-8593-f2d54392a0e8" (UID: "0f872428-e065-46fd-8593-f2d54392a0e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:36:20.937814 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.937770 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:20.953752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953710 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.953752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953751 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953767 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953784 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f872428-e065-46fd-8593-f2d54392a0e8-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953801 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvjn\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-kube-api-access-dqvjn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953817 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f872428-e065-46fd-8593-f2d54392a0e8-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953834 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f872428-e065-46fd-8593-f2d54392a0e8-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:20.954071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:20.953849 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f872428-e065-46fd-8593-f2d54392a0e8-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:36:21.304165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.304125 4920 generic.go:356] "Generic (PLEG): container finished" podID="0f872428-e065-46fd-8593-f2d54392a0e8" containerID="53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327" exitCode=0 Oct 30 09:36:21.304433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.304195 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" Oct 30 09:36:21.304433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.304351 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" event={"ID":"0f872428-e065-46fd-8593-f2d54392a0e8","Type":"ContainerDied","Data":"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327"} Oct 30 09:36:21.304554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.304434 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7" event={"ID":"0f872428-e065-46fd-8593-f2d54392a0e8","Type":"ContainerDied","Data":"9948f868f7d3efe324f0469791c9f9f58ae0ec2cb3eaf87d1c15b6dbcebdfdae"} Oct 30 09:36:21.304554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.304468 4920 scope.go:122] "RemoveContainer" containerID="53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327" Oct 30 09:36:21.318728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.318696 4920 scope.go:122] "RemoveContainer" containerID="53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327" Oct 30 09:36:21.319095 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:36:21.319076 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327\": container with ID starting with 53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327 not found: ID does not exist" containerID="53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327" Oct 30 09:36:21.319175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.319102 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327"} err="failed to get container status \"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327\": rpc error: code = NotFound desc = could not find container \"53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327\": container with ID starting with 53c8a433e46977b43f84e5b04ef8ff16e3bbb693973988d8c909a2d2c2f39327 not found: ID does not exist" Oct 30 09:36:21.368671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.368638 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:36:21.380421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:21.380393 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d3047c-openshift-default-cbc4479ff-bpjn7"] Oct 30 09:36:22.363174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:22.363109 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:22.398635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:22.398599 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f872428-e065-46fd-8593-f2d54392a0e8" path="/var/lib/kubelet/pods/0f872428-e065-46fd-8593-f2d54392a0e8/volumes" Oct 30 09:36:23.931394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:23.931346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:25.424607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:25.424571 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:26.968888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:26.968853 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:28.614845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:28.614808 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:30.350839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:30.350805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:31.138951 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:36:31.138917 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:36:31.964043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:31.964013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:33.554953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:33.554912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:35.004500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:35.004465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:36.565199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:36.565169 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:38.067957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:38.067908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:39.571660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:39.571619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:41.054806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:41.054755 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:42.646177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:42.646141 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:44.124980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:44.124942 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:45.595705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:45.595665 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:46.411613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:46.411579 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:36:47.092486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:47.092450 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:48.847999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:48.847950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:50.555384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:50.555348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:51.355162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:51.355127 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:52.057028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:52.056991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:53.663190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:53.663157 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:55.244951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:55.244908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:56.716343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:56.716306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:57.732389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.732353 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:57.856934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.856894 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11378-3b28d5"] Oct 30 09:36:57.857552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857536 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:36:57.857609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857554 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:36:57.857609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857586 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0f872428-e065-46fd-8593-f2d54392a0e8" containerName="istio-proxy" Oct 30 09:36:57.857609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857591 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f872428-e065-46fd-8593-f2d54392a0e8" containerName="istio-proxy" Oct 30 09:36:57.857713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857677 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:36:57.857713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857686 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:36:57.857713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.857698 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f872428-e065-46fd-8593-f2d54392a0e8" containerName="istio-proxy" Oct 30 09:36:57.861281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.861262 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:36:57.884995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.884954 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:36:57.891808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.891776 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-3b28d5"] Oct 30 09:36:57.962467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:57.962428 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7jk\" (UniqueName: \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\") pod \"test-curl-11378-3b28d5\" (UID: \"4f360a50-db29-472b-9b6b-47a799c14485\") " pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:36:58.064388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.064345 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7jk\" (UniqueName: \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\") pod \"test-curl-11378-3b28d5\" (UID: \"4f360a50-db29-472b-9b6b-47a799c14485\") " pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:36:58.083854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.083825 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7jk\" (UniqueName: \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\") pod \"test-curl-11378-3b28d5\" (UID: \"4f360a50-db29-472b-9b6b-47a799c14485\") " pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:36:58.175658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.175620 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:36:58.347772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.347678 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:36:58.494390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.494350 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-3b28d5"] Oct 30 09:36:58.498539 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:36:58.497601 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f360a50_db29_472b_9b6b_47a799c14485.slice/crio-e0810c0857b164a79325d40c970e622d7f0778caee74f51dab621bcc058d5108 WatchSource:0}: Error finding container e0810c0857b164a79325d40c970e622d7f0778caee74f51dab621bcc058d5108: Status 404 returned error can't find the container with id e0810c0857b164a79325d40c970e622d7f0778caee74f51dab621bcc058d5108 Oct 30 09:36:58.500458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.500437 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:36:58.645062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:58.644972 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11378-3b28d5" event={"ID":"4f360a50-db29-472b-9b6b-47a799c14485","Type":"ContainerStarted","Data":"e0810c0857b164a79325d40c970e622d7f0778caee74f51dab621bcc058d5108"} Oct 30 09:36:59.865056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:36:59.865017 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:00.708955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:00.708908 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11378-3b28d5" podStartSLOduration=1.6363559300000001 podStartE2EDuration="3.708895948s" totalImagesPullingTime="2.072540018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:00.706772615 +0000 UTC m=+2584.430199778" watchObservedRunningTime="2026-10-30 09:37:00.708895948 +0000 UTC m=+2584.432323090" Oct 30 09:37:01.140362 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:01.140299 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:37:01.457496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:01.457406 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:01.592811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:01.591842 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11378-3b28d5"] Oct 30 09:37:01.596030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:01.596000 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11378-3b28d5"] Oct 30 09:37:01.680863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:01.680823 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11378-3b28d5" podUID="4f360a50-db29-472b-9b6b-47a799c14485" containerName="test-curl-11378-3b28d5" containerID="cri-o://1998bfde5792d4c3949f4f9639da1381ed5e5a01140f4fce69497bd0bd2c501e" gracePeriod=2 Oct 30 09:37:01.693927 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:01.693881 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11378-3b28d5\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="4f360a50-db29-472b-9b6b-47a799c14485" pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:37:02.226088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.226046 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:02.583978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.583946 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 09:37:02.584659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.584643 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:37:02.584721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.584661 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5fb028-3125-4c93-98ec-c890f82e3d85" containerName="istio-proxy" Oct 30 09:37:02.584721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.584681 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4f360a50-db29-472b-9b6b-47a799c14485" containerName="test-curl-11378-3b28d5" Oct 30 09:37:02.584721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.584686 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f360a50-db29-472b-9b6b-47a799c14485" containerName="test-curl-11378-3b28d5" Oct 30 09:37:02.584821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.584777 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f360a50-db29-472b-9b6b-47a799c14485" containerName="test-curl-11378-3b28d5" Oct 30 09:37:02.588531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.588506 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.611239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.606503 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8ebd65-openshift-default-dockercfg-dczb8\"" Oct 30 09:37:02.624043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.624008 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 09:37:02.637398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.636760 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:02.638671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638640 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.638838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638711 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.638838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638771 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.638838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638809 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.639036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638854 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.639036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638904 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.639036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.638954 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.639036 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.639009 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.639288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.639037 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz2z\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.652736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.652705 4920 scope.go:122] "RemoveContainer" containerID="6782828345eb65dd7f438d813e83c462fb8f7e42d435a6d3e2406d2908bc8454" Oct 30 09:37:02.663344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.663313 4920 scope.go:122] "RemoveContainer" containerID="7b75ab852e75bfd777048672ebba4057b89140e7f3bb1d4a4af9d2949c7f228d" Oct 30 09:37:02.673347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.673328 4920 scope.go:122] "RemoveContainer" containerID="64ab62cb4d6103deab3d831e1d19acdc4692163bb0950bb83ae7e8513e1bcc2b" Oct 30 09:37:02.770603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.770227 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.770603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.770344 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.770603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.770450 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.770603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.770492 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz2z\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.770603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.770541 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.772073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.771248 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.772073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.771334 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.772073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.771392 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.772073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.771465 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.777265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.773029 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.777265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.773454 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.777265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.774128 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.779562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.777917 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.779562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.778457 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.783935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.782683 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.786692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.784182 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.849281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.848440 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz2z\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.849281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.848902 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\") pod \"e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:02.914282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:02.913571 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:03.057345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.057303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:03.242454 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:03.242409 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98559665_d64b_4d4f_8787_e3788e6cef75.slice/crio-3c8ec112b30b1e57fca7d5e19ff5333b58c0ca35ee9d8fde9ad91c749c313b14 WatchSource:0}: Error finding container 3c8ec112b30b1e57fca7d5e19ff5333b58c0ca35ee9d8fde9ad91c749c313b14: Status 404 returned error can't find the container with id 3c8ec112b30b1e57fca7d5e19ff5333b58c0ca35ee9d8fde9ad91c749c313b14 Oct 30 09:37:03.243490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.243462 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 09:37:03.245591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.245559 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:03.245690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.245625 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:03.245690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.245653 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:03.281134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.281104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:03.885460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.885426 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerStarted","Data":"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba"} Oct 30 09:37:03.885613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.885487 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerStarted","Data":"3c8ec112b30b1e57fca7d5e19ff5333b58c0ca35ee9d8fde9ad91c749c313b14"} Oct 30 09:37:03.886895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.886873 4920 generic.go:356] "Generic (PLEG): container finished" podID="4f360a50-db29-472b-9b6b-47a799c14485" containerID="1998bfde5792d4c3949f4f9639da1381ed5e5a01140f4fce69497bd0bd2c501e" exitCode=137 Oct 30 09:37:03.913931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.913903 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:03.916435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.916408 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:37:03.950652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:03.950544 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podStartSLOduration=1.950525237 podStartE2EDuration="1.950525237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:03.936782114 +0000 UTC m=+2587.660209287" watchObservedRunningTime="2026-10-30 09:37:03.950525237 +0000 UTC m=+2587.673952381" Oct 30 09:37:04.002872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.002833 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\" (UniqueName: \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\") pod \"4f360a50-db29-472b-9b6b-47a799c14485\" (UID: \"4f360a50-db29-472b-9b6b-47a799c14485\") " Oct 30 09:37:04.005318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.005288 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk" pod "4f360a50-db29-472b-9b6b-47a799c14485" (UID: "4f360a50-db29-472b-9b6b-47a799c14485"). InnerVolumeSpecName "kube-api-access-jg7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:04.106085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.106042 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7jk\" (UniqueName: \"kubernetes.io/projected/4f360a50-db29-472b-9b6b-47a799c14485-kube-api-access-jg7jk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:04.398660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.398619 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f360a50-db29-472b-9b6b-47a799c14485" path="/var/lib/kubelet/pods/4f360a50-db29-472b-9b6b-47a799c14485/volumes" Oct 30 09:37:04.831276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.831239 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:04.934437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.933585 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-3b28d5" Oct 30 09:37:04.934437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.933910 4920 scope.go:122] "RemoveContainer" containerID="1998bfde5792d4c3949f4f9639da1381ed5e5a01140f4fce69497bd0bd2c501e" Oct 30 09:37:04.935801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:04.935779 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:05.944817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:05.944786 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:05.945691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:05.945673 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:37:06.315594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:06.315561 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:06.412259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:06.412226 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:06.512618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:06.512580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:08.032751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:08.032716 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:08.134045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:08.134004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:08.248829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:08.248790 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:09.864198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:09.864163 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:11.034690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.034654 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb"] Oct 30 09:37:11.048020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.047986 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d55f4f86-pmrzb"] Oct 30 09:37:11.417842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.417753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:11.749210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.749128 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:37:11.749403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.749364 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" containerID="cri-o://0f7085b948442d6fc089057e2c96cf08b36d514df56fe0b232986ff47b713872" gracePeriod=30 Oct 30 09:37:11.761676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.761645 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:37:11.761948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.761890 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" containerID="cri-o://38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2" gracePeriod=30 Oct 30 09:37:11.767275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.767247 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:37:11.767500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.767474 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" containerID="cri-o://7874f04d6adab9664872587cf430c4b6545554d704259b5c8fc1c29a83ab051e" gracePeriod=30 Oct 30 09:37:11.777312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.777284 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:37:11.777546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:11.777500 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" containerID="cri-o://c4b71cc9bb0bb01b7d9dfb1d26fb68dfff2e09e726a5a1f924fff4551ec277a9" gracePeriod=30 Oct 30 09:37:12.399272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:12.399201 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faecb335-0acc-41a4-80ed-360fba9f1523" path="/var/lib/kubelet/pods/faecb335-0acc-41a4-80ed-360fba9f1523/volumes" Oct 30 09:37:12.928090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:12.928053 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:13.077241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:13.072971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:14.717765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:14.717733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:15.608144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:15.608111 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:16.403880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:16.403838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:17.079822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.079790 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:37:17.090371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.090337 4920 generic.go:356] "Generic (PLEG): container finished" podID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerID="7874f04d6adab9664872587cf430c4b6545554d704259b5c8fc1c29a83ab051e" exitCode=0 Oct 30 09:37:17.091275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.090431 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" event={"ID":"107f7c37-3e30-4b2d-919b-9b0a630e3e60","Type":"ContainerDied","Data":"7874f04d6adab9664872587cf430c4b6545554d704259b5c8fc1c29a83ab051e"} Oct 30 09:37:17.093430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.093403 4920 generic.go:356] "Generic (PLEG): container finished" podID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerID="c4b71cc9bb0bb01b7d9dfb1d26fb68dfff2e09e726a5a1f924fff4551ec277a9" exitCode=0 Oct 30 09:37:17.093542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.093513 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" event={"ID":"28cfc529-10a0-4e9d-b7ff-7c23b346da87","Type":"ContainerDied","Data":"c4b71cc9bb0bb01b7d9dfb1d26fb68dfff2e09e726a5a1f924fff4551ec277a9"} Oct 30 09:37:17.095984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.095959 4920 generic.go:356] "Generic (PLEG): container finished" podID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerID="0f7085b948442d6fc089057e2c96cf08b36d514df56fe0b232986ff47b713872" exitCode=0 Oct 30 09:37:17.096153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.096130 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" event={"ID":"f85f20d6-08cb-4112-8200-c84d9abf97f5","Type":"ContainerDied","Data":"0f7085b948442d6fc089057e2c96cf08b36d514df56fe0b232986ff47b713872"} Oct 30 09:37:17.097904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.097879 4920 generic.go:356] "Generic (PLEG): container finished" podID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerID="38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2" exitCode=0 Oct 30 09:37:17.098073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.097925 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" event={"ID":"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa","Type":"ContainerDied","Data":"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2"} Oct 30 09:37:17.098073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.097963 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" event={"ID":"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa","Type":"ContainerDied","Data":"a609a9379a934bc4f200a772a47fea3d7911b80f7a534a0ab1b9fbd46e3ec330"} Oct 30 09:37:17.098073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.097991 4920 scope.go:122] "RemoveContainer" containerID="38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2" Oct 30 09:37:17.098269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.098150 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk" Oct 30 09:37:17.113679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.113476 4920 scope.go:122] "RemoveContainer" containerID="38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2" Oct 30 09:37:17.114006 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:17.113979 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2\": container with ID starting with 38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2 not found: ID does not exist" containerID="38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2" Oct 30 09:37:17.114092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.114037 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2"} err="failed to get container status \"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2\": rpc error: code = NotFound desc = could not find container \"38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2\": container with ID starting with 38371fc088210da2a7b76c551505bf5bc6cd4555953cc0e68b5ab82ba8ec42b2 not found: ID does not exist" Oct 30 09:37:17.191321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.191287 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:37:17.194072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194033 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194189 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194303 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.194399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194352 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194414 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194483 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194516 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194566 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194614 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194707 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\") pod \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\" (UID: \"1b2e19e1-79b1-4bb2-9bab-682aaa7599aa\") " Oct 30 09:37:17.194810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194752 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.194897 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.194875 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.195025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195004 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.195235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195197 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:17.195420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195397 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.195505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195428 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.195505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195444 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.195505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195458 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.195505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.195472 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.197579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.197543 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.198119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.198083 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.198253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.198235 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:17.199336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.199307 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2" pod "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" (UID: "1b2e19e1-79b1-4bb2-9bab-682aaa7599aa"). InnerVolumeSpecName "kube-api-access-lzmq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.202160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.202144 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:37:17.205018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.204998 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298157 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298247 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298295 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298324 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298364 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298402 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298455 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298496 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298586 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298633 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298675 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298752 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298795 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298830 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298897 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.298940 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299011 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299065 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299154 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299194 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299262 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299360 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299405 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\") pod \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\" (UID: \"107f7c37-3e30-4b2d-919b-9b0a630e3e60\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299466 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299519 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299602 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\") pod \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\" (UID: \"28cfc529-10a0-4e9d-b7ff-7c23b346da87\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299660 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\") pod \"f85f20d6-08cb-4112-8200-c84d9abf97f5\" (UID: \"f85f20d6-08cb-4112-8200-c84d9abf97f5\") " Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.299751 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.300635 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzmq2\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-kube-api-access-lzmq2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.300668 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.300685 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.300701 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.300998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.300717 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.303171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.302328 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.304548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.304496 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:17.305136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.304829 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:17.306554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.306496 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:17.306764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.306719 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.306964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.306937 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.307228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.307136 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.307459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.307427 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.307832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.307799 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:17.308067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.308042 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.309063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.309023 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "kube-api-access-b5mnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.310088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310061 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "kube-api-access-mbvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.310434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310200 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.310434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310317 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "kube-api-access-464p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.310434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310337 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.310647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310626 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.310705 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310654 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.310852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310833 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.310914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.310844 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.311336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.311306 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:17.311336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.311315 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.313173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.313145 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.319650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.319604 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:17.320942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.320093 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token" pod "107f7c37-3e30-4b2d-919b-9b0a630e3e60" (UID: "107f7c37-3e30-4b2d-919b-9b0a630e3e60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.323816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.323788 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy" pod "28cfc529-10a0-4e9d-b7ff-7c23b346da87" (UID: "28cfc529-10a0-4e9d-b7ff-7c23b346da87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:17.323922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.323821 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token" pod "f85f20d6-08cb-4112-8200-c84d9abf97f5" (UID: "f85f20d6-08cb-4112-8200-c84d9abf97f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:17.401504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401470 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401501 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401512 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvb4\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-kube-api-access-mbvb4\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401521 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401529 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401537 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401545 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464p9\" (UniqueName: \"kubernetes.io/projected/28cfc529-10a0-4e9d-b7ff-7c23b346da87-kube-api-access-464p9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401555 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401564 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401572 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401580 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401589 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401597 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401606 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401613 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401621 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401628 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401635 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85f20d6-08cb-4112-8200-c84d9abf97f5-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401643 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401650 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/107f7c37-3e30-4b2d-919b-9b0a630e3e60-kube-api-access-b5mnv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401657 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401665 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401676 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cfc529-10a0-4e9d-b7ff-7c23b346da87-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401688 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107f7c37-3e30-4b2d-919b-9b0a630e3e60-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401699 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85f20d6-08cb-4112-8200-c84d9abf97f5-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.401807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.401713 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85f20d6-08cb-4112-8200-c84d9abf97f5-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:17.445336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.445298 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:37:17.449625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:17.449597 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-tf9dk"] Oct 30 09:37:18.109384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.109349 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" event={"ID":"107f7c37-3e30-4b2d-919b-9b0a630e3e60","Type":"ContainerDied","Data":"14a810a0c9964ad43f64bb785a64bd5607bbae9e8b23f59acf4db10f06787ca3"} Oct 30 09:37:18.109384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.109384 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v" Oct 30 09:37:18.109729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.109418 4920 scope.go:122] "RemoveContainer" containerID="7874f04d6adab9664872587cf430c4b6545554d704259b5c8fc1c29a83ab051e" Oct 30 09:37:18.111938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.111853 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" Oct 30 09:37:18.111938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.111863 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g" event={"ID":"28cfc529-10a0-4e9d-b7ff-7c23b346da87","Type":"ContainerDied","Data":"7ac05789bb4a5118899e0da740db3ffc854481293666890794a6e6fb3b57b856"} Oct 30 09:37:18.113834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.113814 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" Oct 30 09:37:18.113965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.113841 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9" event={"ID":"f85f20d6-08cb-4112-8200-c84d9abf97f5","Type":"ContainerDied","Data":"b8e77b25e13f3d2de73a615a0a74206d789e5be87fe237b79bd2120567941e36"} Oct 30 09:37:18.125538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.125513 4920 scope.go:122] "RemoveContainer" containerID="c4b71cc9bb0bb01b7d9dfb1d26fb68dfff2e09e726a5a1f924fff4551ec277a9" Oct 30 09:37:18.137660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.137640 4920 scope.go:122] "RemoveContainer" containerID="0f7085b948442d6fc089057e2c96cf08b36d514df56fe0b232986ff47b713872" Oct 30 09:37:18.152696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.152665 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:18.158669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.158642 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11390-bc19eb"] Oct 30 09:37:18.159334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159314 4920 cpu_manager.go:414] "containerMap: removing container" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" Oct 30 09:37:18.159334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159333 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159349 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159355 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159368 4920 cpu_manager.go:414] "containerMap: removing container" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159373 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159401 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" Oct 30 09:37:18.159528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159409 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" Oct 30 09:37:18.159803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159536 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" containerName="istio-proxy" Oct 30 09:37:18.159803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159549 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" containerName="istio-proxy" Oct 30 09:37:18.159803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159563 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" containerName="istio-proxy" Oct 30 09:37:18.159803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.159570 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" containerName="istio-proxy" Oct 30 09:37:18.164722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.164704 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:18.174774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.174743 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:37:18.182108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.182074 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:37:18.187525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.187494 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kfpz9"] Oct 30 09:37:18.192342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.192311 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-bc19eb"] Oct 30 09:37:18.241524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.241489 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwb5\" (UniqueName: \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\") pod \"test-curl-11390-bc19eb\" (UID: \"0b47f550-09a9-4ad0-bedb-129c07cb364a\") " pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:18.254267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.250274 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:37:18.254267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.253078 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-pv94v"] Oct 30 09:37:18.289365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.289325 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:37:18.293917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.293886 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-t6n5g"] Oct 30 09:37:18.342886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.342853 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwb5\" (UniqueName: \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\") pod \"test-curl-11390-bc19eb\" (UID: \"0b47f550-09a9-4ad0-bedb-129c07cb364a\") " pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:18.361661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.361590 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwb5\" (UniqueName: \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\") pod \"test-curl-11390-bc19eb\" (UID: \"0b47f550-09a9-4ad0-bedb-129c07cb364a\") " pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:18.399484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.399446 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107f7c37-3e30-4b2d-919b-9b0a630e3e60" path="/var/lib/kubelet/pods/107f7c37-3e30-4b2d-919b-9b0a630e3e60/volumes" Oct 30 09:37:18.400269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.400250 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2e19e1-79b1-4bb2-9bab-682aaa7599aa" path="/var/lib/kubelet/pods/1b2e19e1-79b1-4bb2-9bab-682aaa7599aa/volumes" Oct 30 09:37:18.400889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.400873 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cfc529-10a0-4e9d-b7ff-7c23b346da87" path="/var/lib/kubelet/pods/28cfc529-10a0-4e9d-b7ff-7c23b346da87/volumes" Oct 30 09:37:18.401692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.401675 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85f20d6-08cb-4112-8200-c84d9abf97f5" path="/var/lib/kubelet/pods/f85f20d6-08cb-4112-8200-c84d9abf97f5/volumes" Oct 30 09:37:18.480026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.479989 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:18.798304 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:18.798265 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b47f550_09a9_4ad0_bedb_129c07cb364a.slice/crio-5483a515f4f348c2780c4406f48343412f7db1030687cc404a64b5b872adc8e2 WatchSource:0}: Error finding container 5483a515f4f348c2780c4406f48343412f7db1030687cc404a64b5b872adc8e2: Status 404 returned error can't find the container with id 5483a515f4f348c2780c4406f48343412f7db1030687cc404a64b5b872adc8e2 Oct 30 09:37:18.799447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:18.799425 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-bc19eb"] Oct 30 09:37:19.123869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.123784 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-bc19eb" event={"ID":"0b47f550-09a9-4ad0-bedb-129c07cb364a","Type":"ContainerStarted","Data":"4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8"} Oct 30 09:37:19.123869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.123823 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-bc19eb" event={"ID":"0b47f550-09a9-4ad0-bedb-129c07cb364a","Type":"ContainerStarted","Data":"5483a515f4f348c2780c4406f48343412f7db1030687cc404a64b5b872adc8e2"} Oct 30 09:37:19.161685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.161633 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11390-bc19eb" podStartSLOduration=1.161620006 podStartE2EDuration="1.161620006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:19.158391998 +0000 UTC m=+2602.881819141" watchObservedRunningTime="2026-10-30 09:37:19.161620006 +0000 UTC m=+2602.885047149" Oct 30 09:37:19.832303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.832269 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:19.995964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.995925 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11390-bc19eb"] Oct 30 09:37:20.000024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:19.999992 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11390-bc19eb"] Oct 30 09:37:20.151983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.151889 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11390-c32e14"] Oct 30 09:37:20.153688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.153656 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" containerName="test-curl-11390-bc19eb" Oct 30 09:37:20.153861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.153846 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" containerName="test-curl-11390-bc19eb" Oct 30 09:37:20.154293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.154274 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" containerName="test-curl-11390-bc19eb" Oct 30 09:37:20.160455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.160432 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:20.175300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.174469 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-c32e14"] Oct 30 09:37:20.279035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.279000 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhv8\" (UniqueName: \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\") pod \"test-curl-11390-c32e14\" (UID: \"2dcb4cc8-4602-4f3e-891f-3aded10f2db1\") " pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:20.380286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.380210 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhv8\" (UniqueName: \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\") pod \"test-curl-11390-c32e14\" (UID: \"2dcb4cc8-4602-4f3e-891f-3aded10f2db1\") " pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:20.400265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.400194 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhv8\" (UniqueName: \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\") pod \"test-curl-11390-c32e14\" (UID: \"2dcb4cc8-4602-4f3e-891f-3aded10f2db1\") " pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:20.487750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.487652 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:20.788760 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:20.788724 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dcb4cc8_4602_4f3e_891f_3aded10f2db1.slice/crio-982182adb6f2e6170ff44ffafd20faa98c603b84fca374431e624a9db891c3f5 WatchSource:0}: Error finding container 982182adb6f2e6170ff44ffafd20faa98c603b84fca374431e624a9db891c3f5: Status 404 returned error can't find the container with id 982182adb6f2e6170ff44ffafd20faa98c603b84fca374431e624a9db891c3f5 Oct 30 09:37:20.788962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:20.788945 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-c32e14"] Oct 30 09:37:21.142282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:21.142171 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-c32e14" event={"ID":"2dcb4cc8-4602-4f3e-891f-3aded10f2db1","Type":"ContainerStarted","Data":"b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e"} Oct 30 09:37:21.142282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:21.142237 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-c32e14" event={"ID":"2dcb4cc8-4602-4f3e-891f-3aded10f2db1","Type":"ContainerStarted","Data":"982182adb6f2e6170ff44ffafd20faa98c603b84fca374431e624a9db891c3f5"} Oct 30 09:37:21.142789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:21.142344 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11390-bc19eb" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" containerName="test-curl-11390-bc19eb" containerID="cri-o://4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8" gracePeriod=2 Oct 30 09:37:21.184279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:21.184227 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11390-c32e14" podStartSLOduration=1.184201171 podStartE2EDuration="1.184201171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:21.179547757 +0000 UTC m=+2604.902974903" watchObservedRunningTime="2026-10-30 09:37:21.184201171 +0000 UTC m=+2604.907628314" Oct 30 09:37:21.453630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:21.453532 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:22.016455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.016408 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11390-c32e14"] Oct 30 09:37:22.022172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.022139 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11390-c32e14"] Oct 30 09:37:22.169513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.169481 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11390-5c41f2"] Oct 30 09:37:22.170127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.170110 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" containerName="test-curl-11390-c32e14" Oct 30 09:37:22.170185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.170128 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" containerName="test-curl-11390-c32e14" Oct 30 09:37:22.170262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.170250 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" containerName="test-curl-11390-c32e14" Oct 30 09:37:22.173835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.173817 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:22.188865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.188842 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-5c41f2"] Oct 30 09:37:22.204643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.204613 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb425\" (UniqueName: \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\") pod \"test-curl-11390-5c41f2\" (UID: \"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4\") " pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:22.305892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.305847 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb425\" (UniqueName: \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\") pod \"test-curl-11390-5c41f2\" (UID: \"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4\") " pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:22.324202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.324165 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb425\" (UniqueName: \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\") pod \"test-curl-11390-5c41f2\" (UID: \"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4\") " pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:22.487517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.487484 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:22.790414 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:22.790379 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaa0e1f6_9cae_4fa7_8c8e_72910e5fe1e4.slice/crio-c83b9f497949128fb19a347b27aaf2d3e2bac4256ad52720e9482e41d67bc61c WatchSource:0}: Error finding container c83b9f497949128fb19a347b27aaf2d3e2bac4256ad52720e9482e41d67bc61c: Status 404 returned error can't find the container with id c83b9f497949128fb19a347b27aaf2d3e2bac4256ad52720e9482e41d67bc61c Oct 30 09:37:22.791675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:22.791653 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-5c41f2"] Oct 30 09:37:23.060684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.060590 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:23.168405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.167010 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-5c41f2" event={"ID":"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4","Type":"ContainerStarted","Data":"dc60ea16fa5b8246df7f95a3015141f1e7c158f9d307cbb3f33653a90ac6a4ac"} Oct 30 09:37:23.168405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.167073 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-5c41f2" event={"ID":"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4","Type":"ContainerStarted","Data":"c83b9f497949128fb19a347b27aaf2d3e2bac4256ad52720e9482e41d67bc61c"} Oct 30 09:37:23.168405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.167361 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11390-c32e14" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" containerName="test-curl-11390-c32e14" containerID="cri-o://b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e" gracePeriod=2 Oct 30 09:37:23.205818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.205769 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11390-5c41f2" podStartSLOduration=1.205753688 podStartE2EDuration="1.205753688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:23.204083448 +0000 UTC m=+2606.927510625" watchObservedRunningTime="2026-10-30 09:37:23.205753688 +0000 UTC m=+2606.929180830" Oct 30 09:37:23.387759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.387725 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:23.427572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.427523 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\" (UniqueName: \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\") pod \"0b47f550-09a9-4ad0-bedb-129c07cb364a\" (UID: \"0b47f550-09a9-4ad0-bedb-129c07cb364a\") " Oct 30 09:37:23.429754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.429723 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5" pod "0b47f550-09a9-4ad0-bedb-129c07cb364a" (UID: "0b47f550-09a9-4ad0-bedb-129c07cb364a"). InnerVolumeSpecName "kube-api-access-bpwb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:23.529910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.529870 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwb5\" (UniqueName: \"kubernetes.io/projected/0b47f550-09a9-4ad0-bedb-129c07cb364a-kube-api-access-bpwb5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:23.995178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.995143 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11390-5c41f2"] Oct 30 09:37:23.998874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:23.998827 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11390-5c41f2"] Oct 30 09:37:24.151549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.151510 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11390-7b5a70"] Oct 30 09:37:24.152130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.152115 4920 cpu_manager.go:414] "containerMap: removing container" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" containerName="test-curl-11390-5c41f2" Oct 30 09:37:24.152172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.152131 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" containerName="test-curl-11390-5c41f2" Oct 30 09:37:24.152275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.152264 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" containerName="test-curl-11390-5c41f2" Oct 30 09:37:24.155728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.155712 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:24.175064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.175026 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-7b5a70"] Oct 30 09:37:24.176780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.176750 4920 generic.go:356] "Generic (PLEG): container finished" podID="0b47f550-09a9-4ad0-bedb-129c07cb364a" containerID="4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8" exitCode=137 Oct 30 09:37:24.176933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.176816 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:24.176933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.176839 4920 scope.go:122] "RemoveContainer" containerID="4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8" Oct 30 09:37:24.195379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.195354 4920 scope.go:122] "RemoveContainer" containerID="4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8" Oct 30 09:37:24.195882 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:24.195852 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8\": container with ID starting with 4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8 not found: ID does not exist" containerID="4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8" Oct 30 09:37:24.196001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.195895 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8"} err="failed to get container status \"4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8\": rpc error: code = NotFound desc = could not find container \"4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8\": container with ID starting with 4052a957d21c4dad346f39d6bff37e5b082ba70d36eb1e89b412d4cfa0fb78a8 not found: ID does not exist" Oct 30 09:37:24.206151 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:24.206123 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-bc19eb\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" pod="opendatahub/test-curl-11390-bc19eb" Oct 30 09:37:24.344933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.344899 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85br\" (UniqueName: \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\") pod \"test-curl-11390-7b5a70\" (UID: \"8852d6c8-4882-4d8e-9d34-2015abad1f94\") " pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:24.399783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.399745 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b47f550-09a9-4ad0-bedb-129c07cb364a" path="/var/lib/kubelet/pods/0b47f550-09a9-4ad0-bedb-129c07cb364a/volumes" Oct 30 09:37:24.446132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.446097 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85br\" (UniqueName: \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\") pod \"test-curl-11390-7b5a70\" (UID: \"8852d6c8-4882-4d8e-9d34-2015abad1f94\") " pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:24.464208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.464174 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85br\" (UniqueName: \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\") pod \"test-curl-11390-7b5a70\" (UID: \"8852d6c8-4882-4d8e-9d34-2015abad1f94\") " pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:24.467866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.467839 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:24.562113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.562072 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:24.774667 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:24.774636 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8852d6c8_4882_4d8e_9d34_2015abad1f94.slice/crio-d7cd9f3ab4574b862e30aab48d28e3db91e488f271eda75432bb5dfb40118745 WatchSource:0}: Error finding container d7cd9f3ab4574b862e30aab48d28e3db91e488f271eda75432bb5dfb40118745: Status 404 returned error can't find the container with id d7cd9f3ab4574b862e30aab48d28e3db91e488f271eda75432bb5dfb40118745 Oct 30 09:37:24.775837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:24.775815 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-7b5a70"] Oct 30 09:37:25.186773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.186737 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-7b5a70" event={"ID":"8852d6c8-4882-4d8e-9d34-2015abad1f94","Type":"ContainerStarted","Data":"57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325"} Oct 30 09:37:25.186912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.186783 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-7b5a70" event={"ID":"8852d6c8-4882-4d8e-9d34-2015abad1f94","Type":"ContainerStarted","Data":"d7cd9f3ab4574b862e30aab48d28e3db91e488f271eda75432bb5dfb40118745"} Oct 30 09:37:25.187008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.186990 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11390-5c41f2" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" containerName="test-curl-11390-5c41f2" containerID="cri-o://dc60ea16fa5b8246df7f95a3015141f1e7c158f9d307cbb3f33653a90ac6a4ac" gracePeriod=2 Oct 30 09:37:25.222173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.222117 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11390-7b5a70" podStartSLOduration=1.22209917 podStartE2EDuration="1.22209917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:25.221874387 +0000 UTC m=+2608.945301532" watchObservedRunningTime="2026-10-30 09:37:25.22209917 +0000 UTC m=+2608.945526314" Oct 30 09:37:25.358962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.358935 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:25.361429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.361407 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\" (UniqueName: \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\") pod \"2dcb4cc8-4602-4f3e-891f-3aded10f2db1\" (UID: \"2dcb4cc8-4602-4f3e-891f-3aded10f2db1\") " Oct 30 09:37:25.363463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.363436 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8" pod "2dcb4cc8-4602-4f3e-891f-3aded10f2db1" (UID: "2dcb4cc8-4602-4f3e-891f-3aded10f2db1"). InnerVolumeSpecName "kube-api-access-pqhv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:25.462836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:25.462742 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhv8\" (UniqueName: \"kubernetes.io/projected/2dcb4cc8-4602-4f3e-891f-3aded10f2db1-kube-api-access-pqhv8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:26.039996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.039957 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11390-7b5a70"] Oct 30 09:37:26.045742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.045708 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11390-7b5a70"] Oct 30 09:37:26.132252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.132184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:26.240520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.240460 4920 generic.go:356] "Generic (PLEG): container finished" podID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" containerID="b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e" exitCode=137 Oct 30 09:37:26.241403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.241350 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:26.241403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.241376 4920 scope.go:122] "RemoveContainer" containerID="b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e" Oct 30 09:37:26.257323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.257301 4920 scope.go:122] "RemoveContainer" containerID="b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e" Oct 30 09:37:26.257681 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:26.257656 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e\": container with ID starting with b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e not found: ID does not exist" containerID="b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e" Oct 30 09:37:26.257730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.257695 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e"} err="failed to get container status \"b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e\": rpc error: code = NotFound desc = could not find container \"b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e\": container with ID starting with b5ece2263ea1f35c6c2e2897c489c69b3bf804121963a8f335b2e52afcb2a11e not found: ID does not exist" Oct 30 09:37:26.267667 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:26.267640 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-c32e14\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:26.399291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:26.399164 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" path="/var/lib/kubelet/pods/2dcb4cc8-4602-4f3e-891f-3aded10f2db1/volumes" Oct 30 09:37:26.408043 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:26.408003 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-c32e14\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="2dcb4cc8-4602-4f3e-891f-3aded10f2db1" pod="opendatahub/test-curl-11390-c32e14" Oct 30 09:37:27.251654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.251626 4920 generic.go:356] "Generic (PLEG): container finished" podID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" containerID="dc60ea16fa5b8246df7f95a3015141f1e7c158f9d307cbb3f33653a90ac6a4ac" exitCode=137 Oct 30 09:37:27.252919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.252887 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11390-7b5a70" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" containerName="test-curl-11390-7b5a70" containerID="cri-o://57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325" gracePeriod=2 Oct 30 09:37:27.415252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.415207 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:27.486689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.486580 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\" (UniqueName: \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\") pod \"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4\" (UID: \"faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4\") " Oct 30 09:37:27.494258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.489192 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425" pod "faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" (UID: "faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4"). InnerVolumeSpecName "kube-api-access-rb425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:27.584753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.584704 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:27.588834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.588809 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb425\" (UniqueName: \"kubernetes.io/projected/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4-kube-api-access-rb425\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:27.687169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:27.687125 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:28.008392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.008356 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:37:28.009331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.009306 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" containerName="test-curl-11390-7b5a70" Oct 30 09:37:28.009462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.009332 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" containerName="test-curl-11390-7b5a70" Oct 30 09:37:28.009575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.009544 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" containerName="test-curl-11390-7b5a70" Oct 30 09:37:28.021121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.021082 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.038641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.038576 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:37:28.041843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.041809 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ae795b-openshift-default-dockercfg-67gl5\"" Oct 30 09:37:28.064057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.064015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:28.094182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094143 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094184 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094360 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094406 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094486 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094529 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094548 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094563 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnk8k\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.094699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.094686 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.197082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.197039 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.197439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.197417 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.197553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.197539 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.197729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.197699 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.201295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.201263 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.213280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.201512 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.213670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.213641 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.213898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.213882 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.214012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.213998 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.214124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.214110 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnk8k\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.214667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.201869 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.232240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.219547 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.232240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.202524 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.232240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.205615 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.232240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.219836 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.243239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.234810 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.243239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.234923 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.250242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.245947 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnk8k\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\") pod \"e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.287925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.287899 4920 scope.go:122] "RemoveContainer" containerID="dc60ea16fa5b8246df7f95a3015141f1e7c158f9d307cbb3f33653a90ac6a4ac" Oct 30 09:37:28.288358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.288056 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:28.352780 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:28.352739 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-5c41f2\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" pod="opendatahub/test-curl-11390-5c41f2" Oct 30 09:37:28.361072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.361040 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:28.401506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.401468 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4" path="/var/lib/kubelet/pods/faa0e1f6-9cae-4fa7-8c8e-72910e5fe1e4/volumes" Oct 30 09:37:28.414625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.410533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:28.691839 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:28.691810 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765b68b6_9f92_4c9b_b295_dafb87c8ab4e.slice/crio-1502a9739b281388d76ee48a888b6b112c05097c38ee3ab2cd5cb321c7c462ae WatchSource:0}: Error finding container 1502a9739b281388d76ee48a888b6b112c05097c38ee3ab2cd5cb321c7c462ae: Status 404 returned error can't find the container with id 1502a9739b281388d76ee48a888b6b112c05097c38ee3ab2cd5cb321c7c462ae Oct 30 09:37:28.693865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.693840 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:37:28.694658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.694626 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:28.694710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.694691 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:28.694762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:28.694737 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:29.148774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.148740 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 09:37:29.153061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.153038 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.173609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.173581 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 09:37:29.181146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.181113 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-8ebd65-serving-cert\"" Oct 30 09:37:29.215993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.215957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:29.229593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.229562 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.229778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.229602 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.229778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.229770 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.229977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.229955 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxwh\" (UniqueName: \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.300966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.300141 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerStarted","Data":"62621ad9ea08776842807305264cd1b92e99e891d9dcc44a579b141bc1b1cc9c"} Oct 30 09:37:29.300966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.300206 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerStarted","Data":"1502a9739b281388d76ee48a888b6b112c05097c38ee3ab2cd5cb321c7c462ae"} Oct 30 09:37:29.316159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.316125 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:29.331469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.331434 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.331469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.331471 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.331746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.331579 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.331746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.331729 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxwh\" (UniqueName: \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.332788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.332756 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.334709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.334680 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.337326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.337299 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.344680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.344295 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podStartSLOduration=2.3442785170000002 podStartE2EDuration="2.344278517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:29.341283465 +0000 UTC m=+2613.064710609" watchObservedRunningTime="2026-10-30 09:37:29.344278517 +0000 UTC m=+2613.067705661" Oct 30 09:37:29.351815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.351788 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxwh\" (UniqueName: \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\") pod \"maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.362041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.362012 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:29.368368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.368338 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:29.413264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.413163 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:29.496669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.495467 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:29.497928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.497300 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:29.539497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.535774 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\" (UniqueName: \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\") pod \"8852d6c8-4882-4d8e-9d34-2015abad1f94\" (UID: \"8852d6c8-4882-4d8e-9d34-2015abad1f94\") " Oct 30 09:37:29.551639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.551607 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br" pod "8852d6c8-4882-4d8e-9d34-2015abad1f94" (UID: "8852d6c8-4882-4d8e-9d34-2015abad1f94"). InnerVolumeSpecName "kube-api-access-g85br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:29.638464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.637524 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85br\" (UniqueName: \"kubernetes.io/projected/8852d6c8-4882-4d8e-9d34-2015abad1f94-kube-api-access-g85br\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:29.865074 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:29.865039 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea5bd4d_d46f_4b4e_8227_da9b85c43af2.slice/crio-7e57e48609df085b43b078202a89c32c98500405ed7be9e3a038fae0605f4687 WatchSource:0}: Error finding container 7e57e48609df085b43b078202a89c32c98500405ed7be9e3a038fae0605f4687: Status 404 returned error can't find the container with id 7e57e48609df085b43b078202a89c32c98500405ed7be9e3a038fae0605f4687 Oct 30 09:37:29.867265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.867227 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 09:37:29.941634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:29.941584 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:37:30.365240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.365091 4920 generic.go:356] "Generic (PLEG): container finished" podID="8852d6c8-4882-4d8e-9d34-2015abad1f94" containerID="57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325" exitCode=137 Oct 30 09:37:30.372236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.366447 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:30.372236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.370518 4920 scope.go:122] "RemoveContainer" containerID="57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325" Oct 30 09:37:30.417581 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:30.417532 4920 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-7b5a70\" is forbidden: User \"system:node:ip-10-0-129-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-105.ec2.internal' and this object" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" pod="opendatahub/test-curl-11390-7b5a70" Oct 30 09:37:30.430639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.430600 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8852d6c8-4882-4d8e-9d34-2015abad1f94" path="/var/lib/kubelet/pods/8852d6c8-4882-4d8e-9d34-2015abad1f94/volumes" Oct 30 09:37:30.431454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.431428 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" event={"ID":"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2","Type":"ContainerStarted","Data":"7e57e48609df085b43b078202a89c32c98500405ed7be9e3a038fae0605f4687"} Oct 30 09:37:30.431738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.431707 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:30.431977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.431965 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:37:30.449518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.449419 4920 scope.go:122] "RemoveContainer" containerID="57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325" Oct 30 09:37:30.449941 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:30.449911 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325\": container with ID starting with 57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325 not found: ID does not exist" containerID="57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325" Oct 30 09:37:30.450039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:30.449962 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325"} err="failed to get container status \"57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325\": rpc error: code = NotFound desc = could not find container \"57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325\": container with ID starting with 57479bcab47b45e9e7cc9043e306747fca96e78b63abd66ed0a83171a4983325 not found: ID does not exist" Oct 30 09:37:31.136465 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:31.136421 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:37:31.150527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.150498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:31.431055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.430957 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" event={"ID":"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2","Type":"ContainerStarted","Data":"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4"} Oct 30 09:37:31.431532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.431062 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:31.472743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.472689 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" podStartSLOduration=1.9446520889999999 podStartE2EDuration="2.47267367s" totalImagesPullingTime="528.021581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:31.469450296 +0000 UTC m=+2615.192877456" watchObservedRunningTime="2026-10-30 09:37:31.47267367 +0000 UTC m=+2615.196100874" Oct 30 09:37:31.763599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.763518 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:31.875914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:31.875883 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:32.956005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:32.955970 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:33.554364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:33.554326 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:33.649489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:33.649433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:34.569031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:34.569001 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:36.059712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:36.059680 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:37.444909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:37.444875 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 09:37:37.668451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:37.668414 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:39.171426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:39.171391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:40.028848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:40.028811 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd"] Oct 30 09:37:40.040855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:40.040826 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d23dab0-clmqd"] Oct 30 09:37:40.405584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:40.405550 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524df766-3f38-458c-91af-1e07d7950558" path="/var/lib/kubelet/pods/524df766-3f38-458c-91af-1e07d7950558/volumes" Oct 30 09:37:40.749421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:40.749336 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:42.244079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:42.244038 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:43.780877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:43.780841 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:45.343131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:45.343087 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:46.947506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:46.947466 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:48.921434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:48.921381 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:50.636236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:50.636185 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:51.515239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:51.514007 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:52.354979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:52.354939 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:52.650296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:52.650207 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:37:52.650476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:52.650454 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" containerID="cri-o://6106f1ccf6806f1bbcfdbbb58f3d49839f2e603fbbaf8cd78f29466b126a1644" gracePeriod=30 Oct 30 09:37:53.850834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:53.850793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:54.330359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:54.330321 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:37:54.330594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:54.330568 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" containerID="cri-o://bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6" gracePeriod=30 Oct 30 09:37:55.549318 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.549280 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:37:55.553619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.553590 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.574111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.574079 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:37:55.579770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.579739 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ae795b-serving-cert\"" Oct 30 09:37:55.642040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.641974 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:55.645896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.645859 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.646092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.645929 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56lq\" (UniqueName: \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.646092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.646013 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.646251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.646125 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.727012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.726967 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:55.747177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.747135 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.747427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.747192 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56lq\" (UniqueName: \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.747427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.747389 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.747566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.747488 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.748589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.748554 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.750137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.750106 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.750310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.750286 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.769195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.769161 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56lq\" (UniqueName: \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\") pod \"maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:55.826329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.825950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:55.876006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:55.875967 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:56.192387 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:56.192357 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094edb06_2aff_4715_8a39_86be7ed5746f.slice/crio-cb7a3d59a08cc58bf660e430e535d016496205bc3cb816669b95aadfa682419b WatchSource:0}: Error finding container cb7a3d59a08cc58bf660e430e535d016496205bc3cb816669b95aadfa682419b: Status 404 returned error can't find the container with id cb7a3d59a08cc58bf660e430e535d016496205bc3cb816669b95aadfa682419b Oct 30 09:37:56.194613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:56.194583 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:37:56.769496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:56.769418 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" event={"ID":"094edb06-2aff-4715-8a39-86be7ed5746f","Type":"ContainerStarted","Data":"71f584ede9a414324b9c7678327a61741397a75c241b6ace2b0f547beff4d81b"} Oct 30 09:37:56.769496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:56.769494 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:37:56.769919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:56.769520 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" event={"ID":"094edb06-2aff-4715-8a39-86be7ed5746f","Type":"ContainerStarted","Data":"cb7a3d59a08cc58bf660e430e535d016496205bc3cb816669b95aadfa682419b"} Oct 30 09:37:56.810987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:56.810868 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" podStartSLOduration=1.469021688 podStartE2EDuration="1.810854341s" totalImagesPullingTime="341.832653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:56.807609783 +0000 UTC m=+2640.531036926" watchObservedRunningTime="2026-10-30 09:37:56.810854341 +0000 UTC m=+2640.534281483" Oct 30 09:37:57.193152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.193073 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:57.382776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.382731 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:57.662967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.662932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:57.782323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.782290 4920 generic.go:356] "Generic (PLEG): container finished" podID="780c1b94-75c2-4585-b657-08f3d0481de4" containerID="6106f1ccf6806f1bbcfdbbb58f3d49839f2e603fbbaf8cd78f29466b126a1644" exitCode=0 Oct 30 09:37:57.782757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.782372 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" event={"ID":"780c1b94-75c2-4585-b657-08f3d0481de4","Type":"ContainerDied","Data":"6106f1ccf6806f1bbcfdbbb58f3d49839f2e603fbbaf8cd78f29466b126a1644"} Oct 30 09:37:57.989688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:57.981513 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:37:58.049692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049657 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.049856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049714 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.049856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049763 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.049856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049817 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.049988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049863 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.049988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.049898 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.050737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.050708 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.050988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.050960 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.051309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.051288 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:58.051870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.051843 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.053846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.053820 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.054075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.054054 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:58.090475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.090439 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 09:37:58.091582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.091555 4920 cpu_manager.go:414] "containerMap: removing container" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" Oct 30 09:37:58.091582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.091582 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" Oct 30 09:37:58.091818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.091805 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" containerName="istio-proxy" Oct 30 09:37:58.096473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.096449 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.113738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.113679 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 09:37:58.119660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.119627 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-388653-openshift-default-dockercfg-p7d5c\"" Oct 30 09:37:58.138511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.138474 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:58.151087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.151043 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.151341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.151124 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.151341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.151156 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\") pod \"780c1b94-75c2-4585-b657-08f3d0481de4\" (UID: \"780c1b94-75c2-4585-b657-08f3d0481de4\") " Oct 30 09:37:58.151508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.151372 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.152598 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.152663 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktdj\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.152701 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.152860 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.152942 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153070 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153114 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153333 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153381 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153535 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780c1b94-75c2-4585-b657-08f3d0481de4-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153587 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153602 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153616 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153629 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780c1b94-75c2-4585-b657-08f3d0481de4-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153642 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.153752 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.153655 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780c1b94-75c2-4585-b657-08f3d0481de4-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.154344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.154322 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "kube-api-access-phtgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:58.155935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.154844 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token" pod "780c1b94-75c2-4585-b657-08f3d0481de4" (UID: "780c1b94-75c2-4585-b657-08f3d0481de4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276201 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276291 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276348 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276399 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktdj\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276434 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276529 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276562 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276631 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276665 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276762 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtgh\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-kube-api-access-phtgh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.277327 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.276777 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780c1b94-75c2-4585-b657-08f3d0481de4-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.296247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.278657 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.296247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.278934 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.296247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.281538 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.296247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.282179 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.303523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.303450 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.320256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.318677 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.332262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.329581 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.336273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.335936 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktdj\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.356283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.351847 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\") pod \"e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.381297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.381186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:58.430675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.426630 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:37:58.781086 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:37:58.781051 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60a5b8e_d331_4901_a58a_77885fe3c002.slice/crio-5760a5a0246816cccae42359af6cbd282050d8d263172f2d54e329b43f947b11 WatchSource:0}: Error finding container 5760a5a0246816cccae42359af6cbd282050d8d263172f2d54e329b43f947b11: Status 404 returned error can't find the container with id 5760a5a0246816cccae42359af6cbd282050d8d263172f2d54e329b43f947b11 Oct 30 09:37:58.782732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.782707 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 09:37:58.784491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.784451 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:58.784577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.784545 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:58.784630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.784591 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:58.792805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.792776 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerStarted","Data":"5760a5a0246816cccae42359af6cbd282050d8d263172f2d54e329b43f947b11"} Oct 30 09:37:58.794415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.794392 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" Oct 30 09:37:58.794538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.794426 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8" event={"ID":"780c1b94-75c2-4585-b657-08f3d0481de4","Type":"ContainerDied","Data":"6f20cc8e8eed940484252a58c62cfc013c4a9ef08b4bb23189305580adf20a21"} Oct 30 09:37:58.794538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.794476 4920 scope.go:122] "RemoveContainer" containerID="6106f1ccf6806f1bbcfdbbb58f3d49839f2e603fbbaf8cd78f29466b126a1644" Oct 30 09:37:58.848983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.848949 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:37:58.858872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:58.858845 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-5vtx8"] Oct 30 09:37:59.057939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.057912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:37:59.621060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.620982 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:37:59.699321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699280 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699335 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699370 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699496 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699604 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:59.699732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699620 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699709 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699745 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699806 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.699908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699819 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:59.699908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.699872 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\") pod \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\" (UID: \"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19\") " Oct 30 09:37:59.700132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.700089 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:59.700640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.700517 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.700810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.700792 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.700908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.700895 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.700986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.700622 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:59.701804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.701783 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:59.703751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.703728 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "kube-api-access-gkj2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:59.703858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.703764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:59.704274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.704239 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:59.705637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.705613 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo" pod "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" (UID: "d8c83ecf-2406-4b6f-9b0d-00bf162a3e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:59.802699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802660 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.802699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802697 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.803315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802712 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkj2s\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-kube-api-access-gkj2s\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.803315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802727 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.803315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802747 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.803315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.802761 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:37:59.805512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.805481 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerStarted","Data":"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4"} Oct 30 09:37:59.807531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.807501 4920 generic.go:356] "Generic (PLEG): container finished" podID="98559665-d64b-4d4f-8787-e3788e6cef75" containerID="a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba" exitCode=0 Oct 30 09:37:59.807661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.807568 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerDied","Data":"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba"} Oct 30 09:37:59.807975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.807956 4920 scope.go:122] "RemoveContainer" containerID="a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba" Oct 30 09:37:59.809105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809080 4920 generic.go:356] "Generic (PLEG): container finished" podID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerID="bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6" exitCode=0 Oct 30 09:37:59.809193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809147 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" event={"ID":"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19","Type":"ContainerDied","Data":"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6"} Oct 30 09:37:59.809193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809179 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" event={"ID":"d8c83ecf-2406-4b6f-9b0d-00bf162a3e19","Type":"ContainerDied","Data":"7bb8905c7f552cd9abe715354c4d0f3a78b9fff21cf011454425ff14b9219e1e"} Oct 30 09:37:59.809297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809201 4920 scope.go:122] "RemoveContainer" containerID="bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6" Oct 30 09:37:59.809334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809304 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx" Oct 30 09:37:59.809850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809820 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:59.809922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809905 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:59.809968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.809954 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:37:59.832150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.832122 4920 scope.go:122] "RemoveContainer" containerID="bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6" Oct 30 09:37:59.832539 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:37:59.832508 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6\": container with ID starting with bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6 not found: ID does not exist" containerID="bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6" Oct 30 09:37:59.832627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.832551 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6"} err="failed to get container status \"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6\": rpc error: code = NotFound desc = could not find container \"bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6\": container with ID starting with bec5115ceb94722a9614fe4f82ca09d1664515283ee218bfbc60af8e306889b6 not found: ID does not exist" Oct 30 09:37:59.848470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.848418 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podStartSLOduration=1.848403715 podStartE2EDuration="1.848403715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:59.845633184 +0000 UTC m=+2643.569060328" watchObservedRunningTime="2026-10-30 09:37:59.848403715 +0000 UTC m=+2643.571830858" Oct 30 09:37:59.884433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.884358 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:37:59.889625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:37:59.889599 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-8k2jx"] Oct 30 09:38:00.399962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.399926 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780c1b94-75c2-4585-b657-08f3d0481de4" path="/var/lib/kubelet/pods/780c1b94-75c2-4585-b657-08f3d0481de4/volumes" Oct 30 09:38:00.400464 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.400449 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" path="/var/lib/kubelet/pods/d8c83ecf-2406-4b6f-9b0d-00bf162a3e19/volumes" Oct 30 09:38:00.427183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.427152 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:38:00.433082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.433048 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:38:00.671626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.671544 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:00.835819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.835786 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerStarted","Data":"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d"} Oct 30 09:38:00.837681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.837659 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:38:00.838588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.838565 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:38:00.939419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.927372 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:00.939419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.927434 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:00.939419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:00.927950 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:38:01.154660 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:38:01.154623 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:38:01.920368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:01.920329 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:01.920946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:01.920910 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:02.287254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.287202 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:02.396285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.396249 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:02.789632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.789604 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:38:02.935257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.934495 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:02.935257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.934576 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:02.942275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.938812 4920 scope.go:122] "RemoveContainer" containerID="8f665683f76e8db9ee49d673994e4f08bda200e258c481e83ab2b912496de13c" Oct 30 09:38:02.961116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:02.961096 4920 scope.go:122] "RemoveContainer" containerID="53d99a197a52af98f0d57ea557cf027f3b921e2fed787ea241d00c30368101d0" Oct 30 09:38:03.922241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:03.921640 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:03.922241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:03.921701 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:03.982035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:03.982003 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:04.076325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:04.076290 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:04.166267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:04.166231 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:04.915625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:04.915588 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:04.915925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:04.915895 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:05.693146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:05.693118 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:05.921435 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:05.921400 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 09:38:05.921733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:05.921701 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 09:38:05.946293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:05.944960 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:38:06.821388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:06.821352 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:38:06.921247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:06.921196 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:38:06.922499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:06.922477 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 09:38:07.414694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:07.414660 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:09.057984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:09.057953 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:10.655522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:10.655492 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:12.235405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:12.235352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:12.395409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:12.395372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:13.950805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:13.950752 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:14.171297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.171264 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:38:14.171964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.171937 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" Oct 30 09:38:14.171964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.171957 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" Oct 30 09:38:14.172132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.172068 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c83ecf-2406-4b6f-9b0d-00bf162a3e19" containerName="istio-proxy" Oct 30 09:38:14.177800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.177780 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.196664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.196624 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:38:14.203851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.203767 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-aa2fdf-openshift-default-dockercfg-krds6\"" Oct 30 09:38:14.279296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279261 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279329 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279426 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279476 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279629 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279693 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279783 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279841 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjdp\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.279906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.279874 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.345323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.345290 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:14.381531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381493 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381544 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381581 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjdp\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381600 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381631 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381784 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381840 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381900 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.381979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.381940 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.382186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.382074 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.382186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.382109 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.382358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.382334 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.382413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.382366 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.382702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.382683 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.384140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.384116 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.384251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.384229 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.400934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.400906 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.401071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.401002 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjdp\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.495263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.495160 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:14.737428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.737387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:14.924304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.924269 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:38:14.926592 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:38:14.926561 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32beebe9_cdc6_464b_af7e_50bce18f65f9.slice/crio-c52f37a543f894ecd5d4ee9eac14dfcdcf5d280c84eca22348534f9d38ba05de WatchSource:0}: Error finding container c52f37a543f894ecd5d4ee9eac14dfcdcf5d280c84eca22348534f9d38ba05de: Status 404 returned error can't find the container with id c52f37a543f894ecd5d4ee9eac14dfcdcf5d280c84eca22348534f9d38ba05de Oct 30 09:38:14.930365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.930328 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:14.930451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.930417 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:14.930491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.930458 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:14.965091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:14.965060 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" event={"ID":"32beebe9-cdc6-464b-af7e-50bce18f65f9","Type":"ContainerStarted","Data":"c52f37a543f894ecd5d4ee9eac14dfcdcf5d280c84eca22348534f9d38ba05de"} Oct 30 09:38:15.639560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:15.639525 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:15.974674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:15.974574 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" event={"ID":"32beebe9-cdc6-464b-af7e-50bce18f65f9","Type":"ContainerStarted","Data":"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0"} Oct 30 09:38:16.037706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.037644 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podStartSLOduration=2.03762899 podStartE2EDuration="2.03762899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:16.033823906 +0000 UTC m=+2659.757251078" watchObservedRunningTime="2026-10-30 09:38:16.03762899 +0000 UTC m=+2659.761056132" Oct 30 09:38:16.276452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.276368 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:16.496792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.496761 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:16.521379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.521340 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 09:38:16.521680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.521645 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 09:38:16.760769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.760738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:16.860164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:16.860118 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:17.383619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:17.383584 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:17.496693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:17.496658 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 09:38:17.496907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:17.496724 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 09:38:18.496825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:18.496790 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 09:38:18.497262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:18.496841 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 09:38:18.873427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:18.873390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:19.496563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:19.496527 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 09:38:19.496774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:19.496588 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 09:38:20.113715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.113650 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:20.226674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.226639 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:20.325254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.325188 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:20.496621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.496544 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Oct 30 09:38:20.496621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.496608 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Oct 30 09:38:20.576453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:20.576418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:21.706030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:21.705988 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:21.706982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:21.706960 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:21.708421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:21.708394 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:38:21.840998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:21.840963 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:21.939335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:21.939293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:22.434745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:22.434701 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:24.271963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:24.271931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:24.842537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:24.842499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:26.012891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.012849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:26.213276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.213196 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:26.635453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.635415 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:38:26.639649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.639623 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.673447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.673406 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:38:26.673850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.673831 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d60c68-openshift-default-dockercfg-h5vsv\"" Oct 30 09:38:26.686379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.686346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:26.742467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742432 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742592 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742634 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742764 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs94\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742855 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742810 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742875 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.742947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742925 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.743005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.742951 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.743046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.743004 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.791707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.791650 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:26.844260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.844176 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.868436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.868402 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.889431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.889338 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890256 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs94\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890443 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890536 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890642 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890754 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.890871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890858 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.891253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.890116 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.891915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.891894 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.892257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.892239 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.896786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.896352 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.903302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.896479 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.927245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.919364 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.927245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.919892 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.927523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.927327 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.953033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.952996 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs94\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\") pod \"e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:26.958626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:26.958586 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:27.009778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.009740 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:38:27.010355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.010321 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" containerID="cri-o://cba4633658200bb076c85f10d288bf17326ef498002adbaabd9741f51cc3b5e0" gracePeriod=30 Oct 30 09:38:27.372874 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:38:27.372841 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dd6fe5_7347_46b0_8b0f_5c4f2b7c83b2.slice/crio-7a1248f2406efc493ddd8e22de0af94127cd967d31ccfb4c4a2bc317775ffad2 WatchSource:0}: Error finding container 7a1248f2406efc493ddd8e22de0af94127cd967d31ccfb4c4a2bc317775ffad2: Status 404 returned error can't find the container with id 7a1248f2406efc493ddd8e22de0af94127cd967d31ccfb4c4a2bc317775ffad2 Oct 30 09:38:27.373317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.373170 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:38:27.375689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.375659 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:27.375772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.375723 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:27.375772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.375753 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:27.566531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.566492 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:27.666652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:27.666570 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:28.168052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.168004 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerStarted","Data":"a1749ec4924f4e68a929647e1cee753b7e06a3c5750578a9190edb95c9b46eab"} Oct 30 09:38:28.168497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.168074 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerStarted","Data":"7a1248f2406efc493ddd8e22de0af94127cd967d31ccfb4c4a2bc317775ffad2"} Oct 30 09:38:28.208603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.208556 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podStartSLOduration=2.208540903 podStartE2EDuration="2.208540903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:28.207031143 +0000 UTC m=+2671.930458318" watchObservedRunningTime="2026-10-30 09:38:28.208540903 +0000 UTC m=+2671.931968045" Oct 30 09:38:28.957312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.957192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:28.978193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.978161 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:28.984055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:28.984027 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:29.143909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.143880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:29.175254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.175206 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:29.176212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.176190 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:38:29.248475 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.248306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:29.343875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.343837 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:29.776824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.776779 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:29.901905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:29.901868 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:30.655637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:30.655588 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:30.757355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:30.757318 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:31.136567 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:38:31.136532 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:38:31.270763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:31.270716 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:32.217166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.217132 4920 generic.go:356] "Generic (PLEG): container finished" podID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerID="cba4633658200bb076c85f10d288bf17326ef498002adbaabd9741f51cc3b5e0" exitCode=0 Oct 30 09:38:32.217604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.217203 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" event={"ID":"a8b7e1e5-4d24-4b50-b35e-359d1cceae42","Type":"ContainerDied","Data":"cba4633658200bb076c85f10d288bf17326ef498002adbaabd9741f51cc3b5e0"} Oct 30 09:38:32.351128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.351094 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:38:32.377127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.377095 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:32.438964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.438926 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.438989 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439046 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439081 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439196 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439305 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439357 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439422 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.439574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.439497 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:32.440054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.440029 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.440285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.440206 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:32.440549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.440440 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:32.440864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.440843 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:32.442289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.441948 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:32.442801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.442764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:32.443027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.443004 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "kube-api-access-964fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:32.443789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.443764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:32.472991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.472951 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:32.540683 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.540647 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\") pod \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\" (UID: \"a8b7e1e5-4d24-4b50-b35e-359d1cceae42\") " Oct 30 09:38:32.540952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.540929 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket" pod "a8b7e1e5-4d24-4b50-b35e-359d1cceae42" (UID: "a8b7e1e5-4d24-4b50-b35e-359d1cceae42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:32.541146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541131 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541150 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541160 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541170 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541179 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541190 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964fn\" (UniqueName: \"kubernetes.io/projected/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-kube-api-access-964fn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541198 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:32.541350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:32.541207 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b7e1e5-4d24-4b50-b35e-359d1cceae42-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:38:33.228482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.228444 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" Oct 30 09:38:33.228942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.228445 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7" event={"ID":"a8b7e1e5-4d24-4b50-b35e-359d1cceae42","Type":"ContainerDied","Data":"9f0ab16be77160bdd73534b1c14afb47d27b93afc1be5c451ee8fc7770bc0af3"} Oct 30 09:38:33.228942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.228581 4920 scope.go:122] "RemoveContainer" containerID="cba4633658200bb076c85f10d288bf17326ef498002adbaabd9741f51cc3b5e0" Oct 30 09:38:33.256284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.256245 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:33.280772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.280735 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:38:33.292304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.292276 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-8fgt7"] Oct 30 09:38:33.687567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.687536 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:33.818516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:33.818483 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:34.399700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:34.399651 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" path="/var/lib/kubelet/pods/a8b7e1e5-4d24-4b50-b35e-359d1cceae42/volumes" Oct 30 09:38:35.160676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:35.160640 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:35.341603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:35.341562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:35.435412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:35.435332 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:36.756256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:36.756207 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:38.505974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:38.505940 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:40.083068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:40.083039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:41.661987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:41.661950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:43.214872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:43.214832 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:38:43.254238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:43.253707 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:44.962646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:44.962603 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:46.559976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:46.559941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:48.056838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:48.056793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:49.483864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.483828 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:38:49.484527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.484511 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" Oct 30 09:38:49.484582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.484528 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" Oct 30 09:38:49.484655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.484646 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b7e1e5-4d24-4b50-b35e-359d1cceae42" containerName="istio-proxy" Oct 30 09:38:49.489820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.489792 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.526459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.526427 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-881f83-serving-cert\"" Oct 30 09:38:49.543606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.543577 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:38:49.566427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.566395 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:49.657080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.657046 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.657337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.657094 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpz8r\" (UniqueName: \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.657425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.657385 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.657490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.657429 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.759345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.759083 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.759345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.759149 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz8r\" (UniqueName: \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.759578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.759454 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.759578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.759483 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.760549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.760517 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.762020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.761992 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.762797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.762770 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.781058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.781031 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz8r\" (UniqueName: \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\") pod \"maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:49.846069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:49.845760 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:50.223317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:50.222614 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:38:50.229753 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:38:50.229714 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac433f30_005f_4f0e_94f3_779fb966312d.slice/crio-1858988c232004fee1cba4520c1263bf720e1b03d1c536a92053bc5d63448dba WatchSource:0}: Error finding container 1858988c232004fee1cba4520c1263bf720e1b03d1c536a92053bc5d63448dba: Status 404 returned error can't find the container with id 1858988c232004fee1cba4520c1263bf720e1b03d1c536a92053bc5d63448dba Oct 30 09:38:50.460190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:50.460110 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" event={"ID":"ac433f30-005f-4f0e-94f3-779fb966312d","Type":"ContainerStarted","Data":"1858988c232004fee1cba4520c1263bf720e1b03d1c536a92053bc5d63448dba"} Oct 30 09:38:50.852697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:50.852664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:51.363512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:51.363472 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:51.476884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:51.476844 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" event={"ID":"ac433f30-005f-4f0e-94f3-779fb966312d","Type":"ContainerStarted","Data":"6973ff521114a331ce5bd7d3cd98a8920e147ad73ca9479f62690aae40b8e8d0"} Oct 30 09:38:51.477156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:51.477061 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:51.517148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:51.517086 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" podStartSLOduration=1.9270041500000001 podStartE2EDuration="2.517070301s" totalImagesPullingTime="590.066151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:51.515822481 +0000 UTC m=+2695.239249627" watchObservedRunningTime="2026-10-30 09:38:51.517070301 +0000 UTC m=+2695.240497444" Oct 30 09:38:51.655294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:51.646546 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:53.058885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:53.058841 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:54.602378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:54.602338 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:55.523319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.523283 4920 generic.go:356] "Generic (PLEG): container finished" podID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerID="47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4" exitCode=0 Oct 30 09:38:55.523571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.523344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerDied","Data":"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4"} Oct 30 09:38:55.523904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.523883 4920 scope.go:122] "RemoveContainer" containerID="47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4" Oct 30 09:38:55.526149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.526111 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:55.526295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.526206 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:55.526295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:55.526274 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:56.294851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:56.294810 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:56.540981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:56.540946 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerStarted","Data":"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea"} Oct 30 09:38:57.429169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:57.429138 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:38:57.432339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:57.432308 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 09:38:57.432555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:57.432530 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 09:38:57.493944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:57.493912 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:38:57.869263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:57.869201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:58.428238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:58.427653 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 09:38:58.428238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:58.427721 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 09:38:59.354978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:59.354942 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:38:59.442412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:59.442344 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 09:38:59.442731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:38:59.442686 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 09:39:00.429376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:00.429337 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 09:39:00.429850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:00.429401 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 09:39:00.843753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:00.843723 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:39:00.867258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:00.863151 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:00.957077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:00.957043 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:01.156150 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:39:01.156057 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:39:01.428018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:01.427926 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 09:39:01.428254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:01.428011 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 09:39:02.433140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:02.433112 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:39:02.434523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:02.434502 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 09:39:02.466280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:02.466249 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:03.094541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:03.094510 4920 scope.go:122] "RemoveContainer" containerID="b4f46efa6cfdfaab9d7c3b18a2d1f9cb4c2f1a177982f76daac85e116e3f55ed" Oct 30 09:39:03.104871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:03.104846 4920 scope.go:122] "RemoveContainer" containerID="5a8a65d5356fda01d7b3d07b82db1d10b169327f25f5c6e8fbf35b6f83d45757" Oct 30 09:39:03.311121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:03.311094 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:39:03.399623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:03.399548 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:39:03.460848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:03.460812 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:04.015448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:04.015399 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:04.139211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:04.139164 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:04.241578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:04.241541 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:05.680548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:05.680506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:05.818382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:05.818343 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:06.326931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:06.326895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:06.517335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:06.517299 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:07.349398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:07.349360 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:09.095413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:09.095376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:09.782650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:09.782617 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:09.916006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:09.915968 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:10.710926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:10.710887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:11.494827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:11.494786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:11.594117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:11.594085 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:12.407277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:12.407242 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:13.091405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:13.091368 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:39:13.470928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:13.470843 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:14.222758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:14.222722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:15.130570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:15.130472 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:15.796731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:15.796686 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:16.044181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.044146 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:16.263378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.263287 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:39:16.271111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.271077 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.295891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.295856 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:39:16.329262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.303495 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6d4b7a-openshift-default-dockercfg-lnf6h\"" Oct 30 09:39:16.330668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.330631 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.380249 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381389 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381450 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381537 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381562 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381676 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381730 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381842 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.435241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.381929 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj999\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.485294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.485253 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.485625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.485607 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj999\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.485794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.485772 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.485958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.485933 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.489406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.489372 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.494925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.494881 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.499937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.499906 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.509242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.494300 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.509242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.504764 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.509242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.504892 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.509242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.504950 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.509242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.505422 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.530795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.513847 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.531041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.514572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.552316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.516087 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:16.595870 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.595639 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.596116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.595760 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.659105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.659067 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj999\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.678031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.677993 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\") pod \"e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:16.894509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:16.894470 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:17.248789 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:39:17.248750 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59257381_27fa_487d_b6ea_44311a26584c.slice/crio-d6068ad169c600a0ea94e46f2dd1ff3dcc9606b6797a68998c5cfdac1a50f136 WatchSource:0}: Error finding container d6068ad169c600a0ea94e46f2dd1ff3dcc9606b6797a68998c5cfdac1a50f136: Status 404 returned error can't find the container with id d6068ad169c600a0ea94e46f2dd1ff3dcc9606b6797a68998c5cfdac1a50f136 Oct 30 09:39:17.251726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.251690 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:17.251842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.251763 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:17.251842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.251792 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:17.251946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.251920 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:39:17.761770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.761736 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerStarted","Data":"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1"} Oct 30 09:39:17.762289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.761799 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerStarted","Data":"d6068ad169c600a0ea94e46f2dd1ff3dcc9606b6797a68998c5cfdac1a50f136"} Oct 30 09:39:17.767355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.767329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:17.836603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.836540 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podStartSLOduration=1.836519126 podStartE2EDuration="1.836519126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:17.818698974 +0000 UTC m=+2721.542126148" watchObservedRunningTime="2026-10-30 09:39:17.836519126 +0000 UTC m=+2721.559946272" Oct 30 09:39:17.895033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:17.894991 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:18.909675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:18.909633 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:19.571102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:19.571065 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:19.776400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:19.776369 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:19.777300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:19.777274 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:39:20.050965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:20.050928 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:20.168226 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:20.168184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:21.209365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:21.209327 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:39:21.209778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:21.209564 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" containerID="cri-o://ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670" gracePeriod=30 Oct 30 09:39:21.413633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:21.413597 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:21.844970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:21.844935 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:21.983692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:21.983652 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:23.451620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:23.451582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:24.886239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.877675 4920 generic.go:356] "Generic (PLEG): container finished" podID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerID="a1749ec4924f4e68a929647e1cee753b7e06a3c5750578a9190edb95c9b46eab" exitCode=0 Oct 30 09:39:24.886239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.877804 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerDied","Data":"a1749ec4924f4e68a929647e1cee753b7e06a3c5750578a9190edb95c9b46eab"} Oct 30 09:39:24.886239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.879248 4920 scope.go:122] "RemoveContainer" containerID="a1749ec4924f4e68a929647e1cee753b7e06a3c5750578a9190edb95c9b46eab" Oct 30 09:39:24.896867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.896823 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:24.913240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.910989 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:24.913240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:24.911064 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:25.033883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:25.033848 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:25.887893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:25.887860 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerStarted","Data":"2eb7935079e2f2a28bce0f8739dec61d14ed2b792be6d13ad7a82e2b435350e5"} Oct 30 09:39:25.978776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:25.978741 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:39:26.535298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.534804 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:26.633242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.607462 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:39:26.640179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.640140 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.663749 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.663805 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.663841 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664058 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664139 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664186 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664264 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664303 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.664368 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\") pod \"70c73849-a068-45a9-9ef9-af51b83ea6f7\" (UID: \"70c73849-a068-45a9-9ef9-af51b83ea6f7\") " Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.668826 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.669070 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.669269 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.675253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.674059 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.678056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.678008 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.681508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.681457 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:26.682841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.682493 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:26.688639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.686412 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "kube-api-access-x56c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:26.688639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.688121 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo" pod "70c73849-a068-45a9-9ef9-af51b83ea6f7" (UID: "70c73849-a068-45a9-9ef9-af51b83ea6f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:39:26.766699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766652 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.766699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766693 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c73849-a068-45a9-9ef9-af51b83ea6f7-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.766699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766703 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766712 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56c8\" (UniqueName: \"kubernetes.io/projected/70c73849-a068-45a9-9ef9-af51b83ea6f7-kube-api-access-x56c8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766722 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766732 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766741 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766749 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.767045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.766757 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c73849-a068-45a9-9ef9-af51b83ea6f7-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:39:26.897796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.897700 4920 generic.go:356] "Generic (PLEG): container finished" podID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerID="ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670" exitCode=0 Oct 30 09:39:26.904543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.904513 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" Oct 30 09:39:26.927241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.908821 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" event={"ID":"70c73849-a068-45a9-9ef9-af51b83ea6f7","Type":"ContainerDied","Data":"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670"} Oct 30 09:39:26.927241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.908907 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c" event={"ID":"70c73849-a068-45a9-9ef9-af51b83ea6f7","Type":"ContainerDied","Data":"570af7ab1458620d6182b87d30747935294087765d1d039063851f60e9c48f98"} Oct 30 09:39:26.927241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.908937 4920 scope.go:122] "RemoveContainer" containerID="ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670" Oct 30 09:39:26.946035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.946011 4920 scope.go:122] "RemoveContainer" containerID="ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670" Oct 30 09:39:26.946470 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:39:26.946436 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670\": container with ID starting with ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670 not found: ID does not exist" containerID="ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670" Oct 30 09:39:26.946585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.946476 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670"} err="failed to get container status \"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670\": rpc error: code = NotFound desc = could not find container \"ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670\": container with ID starting with ea73c6fe0aba585c6ddfb23eb0d686afa5bed11af4e8fe2d8819896193978670 not found: ID does not exist" Oct 30 09:39:26.980259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.980210 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 09:39:26.980444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.980304 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 09:39:26.980714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.980697 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:39:26.992640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:26.992605 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-tmw9c"] Oct 30 09:39:27.883512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:27.883478 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:27.978453 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:27.978421 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 09:39:27.978864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:27.978473 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 09:39:28.105242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.104622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:28.354444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.354414 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:39:28.355371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.355346 4920 cpu_manager.go:414] "containerMap: removing container" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" Oct 30 09:39:28.355371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.355370 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" Oct 30 09:39:28.355596 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.355581 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" containerName="istio-proxy" Oct 30 09:39:28.360719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.360697 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.380968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.380937 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:39:28.383882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.383849 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-035d7d-openshift-default-dockercfg-7jxl7\"" Oct 30 09:39:28.395441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395408 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395546 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395587 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395628 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjqz\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395745 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395777 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.395924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.395910 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.396024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.396010 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.396153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.396131 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.434562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.434515 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c73849-a068-45a9-9ef9-af51b83ea6f7" path="/var/lib/kubelet/pods/70c73849-a068-45a9-9ef9-af51b83ea6f7/volumes" Oct 30 09:39:28.435943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.435909 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:28.520244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.516085 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.520244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.516195 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529502 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529646 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529729 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529758 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529803 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjqz\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529858 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.530553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.529883 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.532070 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.532404 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.532641 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.532865 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.533093 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.576340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.549569 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.605325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.598962 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjqz\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.630238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.614168 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.684238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.683860 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\") pod \"e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.684238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.684032 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:28.849263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.848123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:28.978761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.978724 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 09:39:28.979386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:28.979351 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 09:39:29.132758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.132286 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:39:29.135003 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:39:29.134973 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054f2f7e_0462_4f88_8cf1_56888d1d20d0.slice/crio-6283663ab4537aecc7fd8eab25182b0ee26fc0bc2a63229da9a7d7172e1cc82f WatchSource:0}: Error finding container 6283663ab4537aecc7fd8eab25182b0ee26fc0bc2a63229da9a7d7172e1cc82f: Status 404 returned error can't find the container with id 6283663ab4537aecc7fd8eab25182b0ee26fc0bc2a63229da9a7d7172e1cc82f Oct 30 09:39:29.138678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.138643 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:29.138798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.138716 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:29.138798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.138747 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:29.175791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.175762 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:39:29.927238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.926849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:29.966093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.966054 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerStarted","Data":"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda"} Oct 30 09:39:29.971278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.970707 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerStarted","Data":"6283663ab4537aecc7fd8eab25182b0ee26fc0bc2a63229da9a7d7172e1cc82f"} Oct 30 09:39:29.978779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.978746 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 09:39:29.979158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:29.978812 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 09:39:30.062703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.062352 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podStartSLOduration=2.062332499 podStartE2EDuration="2.062332499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:30.023002902 +0000 UTC m=+2733.746430046" watchObservedRunningTime="2026-10-30 09:39:30.062332499 +0000 UTC m=+2733.785759642" Oct 30 09:39:30.685406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.685366 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:30.691337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.691310 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:30.976510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.976431 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:30.977811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.977777 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:39:30.979089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.978501 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 09:39:30.979089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:30.978566 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 09:39:31.169925 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:39:31.169881 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:39:31.773521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:31.773469 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:31.983624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:31.983593 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:39:31.985046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:31.985018 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:39:32.394161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:32.394128 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:32.519883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:32.519842 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:33.577851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:33.577821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:34.330986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:34.330925 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:34.433386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:34.433317 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:35.505299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:35.505235 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:37.365113 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:37.365076 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:38.999682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:38.999596 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:40.782440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:40.782405 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:42.066598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:42.066555 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:42.450661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:42.450562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:43.515854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:43.515814 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:43.919057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:43.919010 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:44.043240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:44.042333 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:44.218649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:44.218559 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:44.301141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:44.301073 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:44.662264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:44.662205 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:44.980563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:44.980457 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:46.061911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:46.061864 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:46.199394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:46.199357 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:46.437895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:46.437809 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:46.727940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:46.727810 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:46.889544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:46.889512 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:47.015300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:47.015186 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:47.461500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:47.461461 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:47.577730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:47.577691 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:47.715668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:47.715575 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:47.815197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:47.815164 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:48.724166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:48.724135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:49.178653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:49.178613 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:39:49.217066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:49.217030 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:50.267816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:50.267783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:50.368053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:50.368022 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:50.511448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:50.511409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:51.753262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:51.753206 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:52.135670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:52.135612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:53.701880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:53.701839 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:55.272983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:55.272948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:56.737980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.737946 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:39:56.743243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.743207 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.758925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.758899 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:39:56.765196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.765152 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6d4b7a-serving-cert\"" Oct 30 09:39:56.816245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.816198 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:56.831155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.831127 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.831319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.831159 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.831319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.831195 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.831396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.831349 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4n2\" (UniqueName: \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.915508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.915474 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:56.933330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.933282 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.934379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.934353 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.938063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.938032 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.938482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.938457 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4n2\" (UniqueName: \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.939037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.934300 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.943010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.942984 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.954245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.953171 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:56.962529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:56.962462 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4n2\" (UniqueName: \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\") pod \"maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:57.065086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:57.065052 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:57.369278 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:39:57.369245 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f65189e_ee53_4cd2_92f3_6f1cfe5fe4cf.slice/crio-af2e24b42208cc8cac9a1f3d2f28ddf600804b521d05ef57cf49be0806960048 WatchSource:0}: Error finding container af2e24b42208cc8cac9a1f3d2f28ddf600804b521d05ef57cf49be0806960048: Status 404 returned error can't find the container with id af2e24b42208cc8cac9a1f3d2f28ddf600804b521d05ef57cf49be0806960048 Oct 30 09:39:57.371051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:57.371026 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:39:58.339028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:58.338977 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" event={"ID":"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf","Type":"ContainerStarted","Data":"d2d5a18d0e3974c25200890a26c3b6da4cad6ed2918f9f2fbe379f3787a95a44"} Oct 30 09:39:58.339583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:58.339089 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:39:58.339583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:58.339116 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" event={"ID":"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf","Type":"ContainerStarted","Data":"af2e24b42208cc8cac9a1f3d2f28ddf600804b521d05ef57cf49be0806960048"} Oct 30 09:39:58.381257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:58.381177 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" podStartSLOduration=1.9653440199999999 podStartE2EDuration="2.381159108s" totalImagesPullingTime="415.815088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:58.376819764 +0000 UTC m=+2762.100246908" watchObservedRunningTime="2026-10-30 09:39:58.381159108 +0000 UTC m=+2762.104586253" Oct 30 09:39:58.443194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:58.443158 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:39:59.991072 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:39:59.991031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:01.141172 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:40:01.141131 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:40:01.463674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:01.463595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:02.932931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:02.932898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:03.140625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:03.140594 4920 scope.go:122] "RemoveContainer" containerID="a0f4e6aabe6b69426537241b57acdc2ab65763850fb08106faf21eced28c34ce" Oct 30 09:40:04.355552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:04.355522 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:40:04.452207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:04.452171 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:06.150376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:06.150338 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:07.868471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:07.868435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:09.568936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:09.568901 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:09.666060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:09.666016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:09.791237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:09.789612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:11.357537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:11.357499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:13.031350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:13.031306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:14.575134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:14.575102 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:15.521376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:15.521321 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:16.045739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:16.045688 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:17.690644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:17.690594 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:19.424751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:19.424709 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:20.333971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:20.333936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:21.144093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:21.144057 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:22.817300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:22.817265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:24.359052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:24.359014 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:24.442991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:24.442943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:24.777299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:24.777204 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:25.497358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:25.497322 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:40:26.348105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:26.348061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:26.547912 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:26.547875 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:27.974616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:27.974581 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:28.074911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.074877 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:28.473096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.473062 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 09:40:28.486180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.486131 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:28.495291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.495258 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 09:40:28.495591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.495573 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678584 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678646 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678769 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678798 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678848 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.678937 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.679011 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.679119 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.682246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.679313 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.805562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.794874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817111 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817183 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817238 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817316 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817342 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817388 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817469 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817537 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.820254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.817610 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.844832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.844791 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.885554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.876724 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.885554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.878890 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.885554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.884486 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.885554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.885071 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.885935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.885667 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.891772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.891743 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f58f32-openshift-default-dockercfg-dq68x\"" Oct 30 09:40:28.898151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.898116 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:28.968732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:28.968698 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:29.064770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.064676 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\") pod \"e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:29.364895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.364805 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:29.762948 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:40:29.762894 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16b04fc_2c0a_40b9_ad91_78ccf37aa54e.slice/crio-fc821dbb02e4f77d845802a521590056815a7202bd1ef474609f4bd4e7316789 WatchSource:0}: Error finding container fc821dbb02e4f77d845802a521590056815a7202bd1ef474609f4bd4e7316789: Status 404 returned error can't find the container with id fc821dbb02e4f77d845802a521590056815a7202bd1ef474609f4bd4e7316789 Oct 30 09:40:29.765206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.765187 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 09:40:29.766667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.766637 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:29.766832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.766816 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:29.766923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.766912 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:29.780366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.780348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:29.983559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:29.983527 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerStarted","Data":"fc821dbb02e4f77d845802a521590056815a7202bd1ef474609f4bd4e7316789"} Oct 30 09:40:30.993015 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:30.992979 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerStarted","Data":"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981"} Oct 30 09:40:31.066275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:31.065583 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podStartSLOduration=3.065518371 podStartE2EDuration="3.065518371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:31.048197468 +0000 UTC m=+2794.771624615" watchObservedRunningTime="2026-10-30 09:40:31.065518371 +0000 UTC m=+2794.788945515" Oct 30 09:40:31.268316 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:40:31.258431 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:40:31.282210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:31.282170 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:31.367476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:31.367445 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:31.389580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:31.389548 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:32.017964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:32.017930 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:32.023723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:32.023697 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:40:32.821567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:32.821476 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:33.419238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:33.418514 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:33.524069 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:33.524026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:34.526638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:34.526600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:35.146545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:35.146306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:35.248771 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:35.248730 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:35.818525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:35.818487 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:40:35.819172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:35.819137 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:40:36.251372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:36.251278 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:37.086868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.086787 4920 generic.go:356] "Generic (PLEG): container finished" podID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerID="ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda" exitCode=0 Oct 30 09:40:37.087522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.086863 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerDied","Data":"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda"} Oct 30 09:40:37.087522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.087330 4920 scope.go:122] "RemoveContainer" containerID="ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda" Oct 30 09:40:37.089127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.089095 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:37.089287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.089181 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:37.089287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.089276 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:37.789630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:37.789596 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:38.096264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.096140 4920 generic.go:356] "Generic (PLEG): container finished" podID="59257381-27fa-487d-b6ea-44311a26584c" containerID="b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1" exitCode=0 Oct 30 09:40:38.096264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.096245 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerDied","Data":"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1"} Oct 30 09:40:38.096836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.096817 4920 scope.go:122] "RemoveContainer" containerID="b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1" Oct 30 09:40:38.101723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.099649 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:38.101723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.099753 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:38.101723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.099813 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:38.126240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.119770 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerStarted","Data":"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279"} Oct 30 09:40:38.684742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.684701 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:40:38.684958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.684935 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:38.685020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:38.684996 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:39.142290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.141848 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerStarted","Data":"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3"} Oct 30 09:40:39.583732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.583698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:39.694245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.690678 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:39.694245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.690767 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:39.895086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.895005 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:40:39.895300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.895276 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 09:40:39.895357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:39.895325 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 09:40:40.685523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:40.685489 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:40.685941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:40.685547 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:40.895796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:40.895758 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 09:40:40.895984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:40.895811 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 09:40:41.339101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.339063 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:41.685047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.684961 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:41.685047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.685011 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:41.895276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.895213 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 09:40:41.895672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.895288 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 09:40:41.970680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:41.970602 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:42.684660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:42.684615 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:42.684851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:42.684673 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:42.896110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:42.896051 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 09:40:42.896542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:42.896151 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 09:40:43.003016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.002725 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:43.584724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.584681 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:43.684808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.684762 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 09:40:43.685080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.684824 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 09:40:43.896824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.896726 4920 patch_prober.go:29] interesting pod/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 09:40:43.897650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:43.897601 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 09:40:44.467842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.467803 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:44.690670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.690637 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:40:44.691119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.691103 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:40:44.691931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.691909 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:40:44.969255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.969179 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:40:44.969742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.969576 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:40:44.970628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:44.970595 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:40:46.061680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:46.061643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:47.509348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:47.509313 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:49.098422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:49.098388 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:50.745128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:50.745088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:51.862972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:51.862937 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:52.410073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:52.410039 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:53.961833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:53.961794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:54.307721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:54.307683 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:55.560974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:55.560943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:55.677250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:55.675800 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:55.865756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:55.865666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:56.166689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:56.166607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:56.321714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:56.321677 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:56.532805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:56.532698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:56.971191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:56.971149 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:57.073652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:57.073620 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:57.448242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:57.440248 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:58.356825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:58.356796 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:40:59.269308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:40:59.269271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:00.947117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:00.947076 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:01.137696 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:41:01.137655 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:41:01.337209 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:01.337167 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:41:02.683972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:02.683931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:04.191470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:04.191440 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:05.841534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:05.841498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:06.297820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:06.297781 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:06.399085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:06.399049 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:07.153844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:07.153795 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:07.282437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:07.282392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:07.645423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:07.645390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:09.081334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:09.081300 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:09.294857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:09.294821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:10.086742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.086709 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:10.308988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.308957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:10.340831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.340754 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb"] Oct 30 09:41:10.346535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.346508 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.365015 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.364986 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:41:10.376983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.376954 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:41:10.383501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.383470 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb"] Oct 30 09:41:10.425843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.425809 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:10.449091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.449055 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.449313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.449114 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrlxt\" (UniqueName: \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.449313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.449267 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.550947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.550901 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.551126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.550972 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrlxt\" (UniqueName: \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.551191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.551145 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.551649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.551627 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.553549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.553525 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.570391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.570365 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrlxt\" (UniqueName: \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\") pod \"maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.661836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.661742 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:10.665350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.665314 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:10.973565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:10.973404 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:11.007826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.007796 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb"] Oct 30 09:41:11.074650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.074615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:11.173175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.173140 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:11.534535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.534492 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" event={"ID":"8b03b18c-e6be-466a-b60a-6408e9b5d60d","Type":"ContainerStarted","Data":"36b7f58631c8c5f98591b1beaee5e3fc6927eb2d4657a10a731ff43bf332fe90"} Oct 30 09:41:11.534766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.534562 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" event={"ID":"8b03b18c-e6be-466a-b60a-6408e9b5d60d","Type":"ContainerStarted","Data":"f961ed590885cfde0338fac97af3228dcd685e24f438bf8d2f01dee4cbc704ed"} Oct 30 09:41:11.574932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.574874 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" podStartSLOduration=1.57485595 podStartE2EDuration="1.57485595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:11.57211021 +0000 UTC m=+2835.295537370" watchObservedRunningTime="2026-10-30 09:41:11.57485595 +0000 UTC m=+2835.298283126" Oct 30 09:41:11.681304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.681266 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:11.783567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:11.783487 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:12.545570 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:12.545535 4920 generic.go:356] "Generic (PLEG): container finished" podID="8b03b18c-e6be-466a-b60a-6408e9b5d60d" containerID="36b7f58631c8c5f98591b1beaee5e3fc6927eb2d4657a10a731ff43bf332fe90" exitCode=0 Oct 30 09:41:12.546006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:12.545617 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" event={"ID":"8b03b18c-e6be-466a-b60a-6408e9b5d60d","Type":"ContainerDied","Data":"36b7f58631c8c5f98591b1beaee5e3fc6927eb2d4657a10a731ff43bf332fe90"} Oct 30 09:41:13.046122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.046086 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 09:41:13.050579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.050556 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.061484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.061442 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6tq\" (UniqueName: \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.061635 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.061561 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.061779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.061761 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.061882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.061830 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.064820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.064760 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f58f32-serving-cert\"" Oct 30 09:41:13.070432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.070406 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 09:41:13.135243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.128260 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:13.162941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.162898 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.163140 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:41:13.163068 4920 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-serving-cert: secret "maas-api-e2e-extmod-b-f58f32-serving-cert" not found Oct 30 09:41:13.163140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.163070 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.163261 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:41:13.163137 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls podName:15fc342a-b869-45c4-9b57-8fce99b62bf6 nodeName:}" failed. No retries permitted until 2026-10-30 09:41:13.663117195 +0000 UTC m=+2837.386544340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls") pod "maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" (UID: "15fc342a-b869-45c4-9b57-8fce99b62bf6") : secret "maas-api-e2e-extmod-b-f58f32-serving-cert" not found Oct 30 09:41:13.163261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.163209 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.163381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.163295 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6tq\" (UniqueName: \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.163861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.163837 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.165538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.165514 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.182561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.182536 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6tq\" (UniqueName: \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.222452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.222420 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:13.335436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.335348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:13.706974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.706877 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.732315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.731689 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:13.968148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:13.967241 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:14.031128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.031097 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:14.130440 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.130399 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\") pod \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " Oct 30 09:41:14.130626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.130551 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\") pod \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " Oct 30 09:41:14.131255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.131181 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca" pod "8b03b18c-e6be-466a-b60a-6408e9b5d60d" (UID: "8b03b18c-e6be-466a-b60a-6408e9b5d60d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:14.131540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.131514 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:14.150054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.150021 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca" pod "8b03b18c-e6be-466a-b60a-6408e9b5d60d" (UID: "8b03b18c-e6be-466a-b60a-6408e9b5d60d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:41:14.245289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.244235 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\" (UniqueName: \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\") pod \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\" (UID: \"8b03b18c-e6be-466a-b60a-6408e9b5d60d\") " Oct 30 09:41:14.245289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.244828 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b03b18c-e6be-466a-b60a-6408e9b5d60d-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:14.310595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.310547 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt" pod "8b03b18c-e6be-466a-b60a-6408e9b5d60d" (UID: "8b03b18c-e6be-466a-b60a-6408e9b5d60d"). InnerVolumeSpecName "kube-api-access-zrlxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:14.339923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.339698 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 09:41:14.342623 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:41:14.342575 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fc342a_b869_45c4_9b57_8fce99b62bf6.slice/crio-9c6e4f5c9f3ff222b83cd81862be3e53113679a8efcf7e98d8a6f7c02c3e8912 WatchSource:0}: Error finding container 9c6e4f5c9f3ff222b83cd81862be3e53113679a8efcf7e98d8a6f7c02c3e8912: Status 404 returned error can't find the container with id 9c6e4f5c9f3ff222b83cd81862be3e53113679a8efcf7e98d8a6f7c02c3e8912 Oct 30 09:41:14.346973 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.346922 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrlxt\" (UniqueName: \"kubernetes.io/projected/8b03b18c-e6be-466a-b60a-6408e9b5d60d-kube-api-access-zrlxt\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:14.570311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.570283 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" Oct 30 09:41:14.570500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.570316 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb" event={"ID":"8b03b18c-e6be-466a-b60a-6408e9b5d60d","Type":"ContainerDied","Data":"f961ed590885cfde0338fac97af3228dcd685e24f438bf8d2f01dee4cbc704ed"} Oct 30 09:41:14.570500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.570366 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f961ed590885cfde0338fac97af3228dcd685e24f438bf8d2f01dee4cbc704ed" Oct 30 09:41:14.571821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.571789 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" event={"ID":"15fc342a-b869-45c4-9b57-8fce99b62bf6","Type":"ContainerStarted","Data":"9c6e4f5c9f3ff222b83cd81862be3e53113679a8efcf7e98d8a6f7c02c3e8912"} Oct 30 09:41:14.826002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:14.825912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:15.580871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:15.580831 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" event={"ID":"15fc342a-b869-45c4-9b57-8fce99b62bf6","Type":"ContainerStarted","Data":"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066"} Oct 30 09:41:15.581073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:15.580911 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:15.625765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:15.625692 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" podStartSLOduration=2.246191803 podStartE2EDuration="2.625673319s" totalImagesPullingTime="379.481516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:15.622824467 +0000 UTC m=+2839.346251610" watchObservedRunningTime="2026-10-30 09:41:15.625673319 +0000 UTC m=+2839.349100463" Oct 30 09:41:16.451211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:16.451173 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:18.025572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:18.025530 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:19.075558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:19.075519 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:19.159952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:19.159898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:19.726244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:19.723293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:19.826240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:19.825757 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:20.889694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:20.889657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:20.987735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:20.987702 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:21.257344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:21.257260 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:21.610245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:21.609473 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 09:41:22.882382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:22.882348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:24.452105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:24.452069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:25.885273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:25.885233 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:25.976295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:25.976255 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:26.063483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:26.063445 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:26.159818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:26.159736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:27.830768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:27.830729 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:29.364632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:29.364589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:30.923056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:30.923015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:31.138044 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:41:31.138005 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:41:32.112426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:32.112391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:32.208582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:32.208549 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:32.486099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:32.485994 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:32.606798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:32.606767 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:32.786560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:32.786528 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:33.953273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:33.953238 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:34.204693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:34.204616 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:35.875378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:35.875323 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:37.443051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:37.443012 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:38.923862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:38.923818 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:40.473531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:40.473475 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:42.049386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:42.049343 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:43.574433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:43.574399 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:45.006263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.006193 4920 generic.go:356] "Generic (PLEG): container finished" podID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerID="62621ad9ea08776842807305264cd1b92e99e891d9dcc44a579b141bc1b1cc9c" exitCode=0 Oct 30 09:41:45.006842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.006268 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerDied","Data":"62621ad9ea08776842807305264cd1b92e99e891d9dcc44a579b141bc1b1cc9c"} Oct 30 09:41:45.006842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.006700 4920 scope.go:122] "RemoveContainer" containerID="62621ad9ea08776842807305264cd1b92e99e891d9dcc44a579b141bc1b1cc9c" Oct 30 09:41:45.008629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.008593 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:41:45.008741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.008688 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:41:45.008741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.008725 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:41:45.048633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.048608 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:45.419727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:45.419690 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:41:46.022438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:46.022394 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerStarted","Data":"be6ebb8670f279cbf4805f4bdd0e79568bea12fc30f300132b681c5c42b8ea47"} Oct 30 09:41:46.361249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:46.361180 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:41:46.362644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:46.362619 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:46.362738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:46.362666 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:46.573157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:46.573126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:47.362387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:47.362348 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:47.362820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:47.362400 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:48.058622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:48.058575 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:48.362032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:48.361951 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:48.362032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:48.362007 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:49.362205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:49.362172 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:49.362645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:49.362257 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:49.585899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:49.585863 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:50.361801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:50.361763 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:50.361985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:50.361845 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:51.076384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:51.076348 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:51.362295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:51.362174 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 30 09:41:51.362295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:51.362265 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 30 09:41:51.977161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:51.977126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:52.125869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:52.125830 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:52.367106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:52.367073 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:41:52.367522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:52.367509 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:41:52.368167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:52.368147 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:41:52.647709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:52.647618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:54.179659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:54.179624 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:54.624785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:54.624747 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:41:55.254031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:55.253977 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:56.192241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:56.188004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:56.528868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:56.528776 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:56.637728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:56.637689 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:56.895430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:56.895396 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:57.116704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.116669 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:57.252716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.252632 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:57.348913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.348875 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:57.842250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.842201 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:41:57.842798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.842488 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" podUID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" containerName="maas-api" containerID="cri-o://d2d5a18d0e3974c25200890a26c3b6da4cad6ed2918f9f2fbe379f3787a95a44" gracePeriod=30 Oct 30 09:41:57.889148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.889106 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:57.989558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:57.989524 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:58.523522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.523482 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:58.535131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.535102 4920 generic.go:356] "Generic (PLEG): container finished" podID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" containerID="d2d5a18d0e3974c25200890a26c3b6da4cad6ed2918f9f2fbe379f3787a95a44" exitCode=0 Oct 30 09:41:58.535834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.535812 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" event={"ID":"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf","Type":"ContainerDied","Data":"d2d5a18d0e3974c25200890a26c3b6da4cad6ed2918f9f2fbe379f3787a95a44"} Oct 30 09:41:58.553090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.553066 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:41:58.625585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.625550 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:58.637896 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.637843 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\") pod \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " Oct 30 09:41:58.638068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.637940 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\") pod \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " Oct 30 09:41:58.638352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.638326 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\" (UniqueName: \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\") pod \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " Oct 30 09:41:58.638561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.638514 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca" pod "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" (UID: "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:58.638822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.638801 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\") pod \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\" (UID: \"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf\") " Oct 30 09:41:58.639740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.639655 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:58.642719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.642675 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2" pod "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" (UID: "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf"). InnerVolumeSpecName "kube-api-access-8b4n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:58.643715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.643690 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls" pod "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" (UID: "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:41:58.648946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.648917 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb"] Oct 30 09:41:58.649136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.648965 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6d4b7a-dm7xb"] Oct 30 09:41:58.656779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.656744 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs" pod "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" (UID: "4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:41:58.726817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.726772 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:58.763239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.745686 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4n2\" (UniqueName: \"kubernetes.io/projected/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-kube-api-access-8b4n2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:58.766031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.766000 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:58.766308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:58.766288 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:41:59.124409 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.124329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:59.328873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.328833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:41:59.571160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.571122 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" event={"ID":"4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf","Type":"ContainerDied","Data":"af2e24b42208cc8cac9a1f3d2f28ddf600804b521d05ef57cf49be0806960048"} Oct 30 09:41:59.571491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.571469 4920 scope.go:122] "RemoveContainer" containerID="d2d5a18d0e3974c25200890a26c3b6da4cad6ed2918f9f2fbe379f3787a95a44" Oct 30 09:41:59.571826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.571808 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424" Oct 30 09:41:59.637810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.637773 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:41:59.641962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:41:59.641929 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6d4b7a-5687cd6b95-wn424"] Oct 30 09:42:00.407241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:00.405199 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" path="/var/lib/kubelet/pods/4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf/volumes" Oct 30 09:42:00.407241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:00.407119 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b03b18c-e6be-466a-b60a-6408e9b5d60d" path="/var/lib/kubelet/pods/8b03b18c-e6be-466a-b60a-6408e9b5d60d/volumes" Oct 30 09:42:00.663283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:00.662368 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:01.149117 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:01.149052 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:42:02.160169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:02.160132 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:03.664019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:03.663986 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:04.813748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:04.813708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:04.917014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:04.916978 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:05.153958 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:05.153870 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:42:05.154182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:05.154152 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" containerID="cri-o://21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3" gracePeriod=30 Oct 30 09:42:05.170924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:05.170891 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:05.327491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:05.327457 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:06.637247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.637203 4920 generic.go:356] "Generic (PLEG): container finished" podID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerID="c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981" exitCode=0 Oct 30 09:42:06.637698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.637293 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerDied","Data":"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981"} Oct 30 09:42:06.637782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.637763 4920 scope.go:122] "RemoveContainer" containerID="c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981" Oct 30 09:42:06.640058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.640023 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:06.640169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.640112 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:06.640169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.640159 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:06.964237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:06.964142 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:07.648388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:07.648349 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerStarted","Data":"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae"} Oct 30 09:42:08.365531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:08.365490 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:42:08.365720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:08.365590 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 09:42:08.365720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:08.365652 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 09:42:08.531362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:08.531289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:09.366408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:09.366372 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 09:42:09.366838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:09.366427 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 09:42:10.084481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.084444 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:10.365891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.365815 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 09:42:10.366046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.365886 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 09:42:10.554120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.554088 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:42:10.633788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.633697 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634118 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634321 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634438 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634609 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634735 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634839 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.634977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.634963 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.635076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.635064 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\") pod \"59257381-27fa-487d-b6ea-44311a26584c\" (UID: \"59257381-27fa-487d-b6ea-44311a26584c\") " Oct 30 09:42:10.636086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.636065 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.638999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.638730 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.639205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.639169 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.642256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.641305 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.642256 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.641989 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:10.642621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.642602 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:10.643687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.643663 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:42:10.644393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.644361 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "kube-api-access-zj999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:10.649394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.649349 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy" pod "59257381-27fa-487d-b6ea-44311a26584c" (UID: "59257381-27fa-487d-b6ea-44311a26584c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:10.688634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.688595 4920 generic.go:356] "Generic (PLEG): container finished" podID="59257381-27fa-487d-b6ea-44311a26584c" containerID="21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3" exitCode=0 Oct 30 09:42:10.688837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.688652 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" Oct 30 09:42:10.688837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.688672 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerDied","Data":"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3"} Oct 30 09:42:10.688837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.688729 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m" event={"ID":"59257381-27fa-487d-b6ea-44311a26584c","Type":"ContainerDied","Data":"d6068ad169c600a0ea94e46f2dd1ff3dcc9606b6797a68998c5cfdac1a50f136"} Oct 30 09:42:10.688837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.688758 4920 scope.go:122] "RemoveContainer" containerID="21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.736986 4920 scope.go:122] "RemoveContainer" containerID="b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737794 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737814 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737827 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737837 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59257381-27fa-487d-b6ea-44311a26584c-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737846 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737856 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737865 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj999\" (UniqueName: \"kubernetes.io/projected/59257381-27fa-487d-b6ea-44311a26584c-kube-api-access-zj999\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737873 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59257381-27fa-487d-b6ea-44311a26584c-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.743253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.737881 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59257381-27fa-487d-b6ea-44311a26584c-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:10.750046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.750012 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:42:10.751976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.751955 4920 scope.go:122] "RemoveContainer" containerID="21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3" Oct 30 09:42:10.752425 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:10.752404 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3\": container with ID starting with 21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3 not found: ID does not exist" containerID="21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3" Oct 30 09:42:10.752489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.752437 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3"} err="failed to get container status \"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3\": rpc error: code = NotFound desc = could not find container \"21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3\": container with ID starting with 21196bac298cae0895499acfa3e7fd7151f451165aba23b7fb6985e9ada7aec3 not found: ID does not exist" Oct 30 09:42:10.752489 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.752457 4920 scope.go:122] "RemoveContainer" containerID="b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1" Oct 30 09:42:10.752737 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:10.752714 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1\": container with ID starting with b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1 not found: ID does not exist" containerID="b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1" Oct 30 09:42:10.752788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.752745 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1"} err="failed to get container status \"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1\": rpc error: code = NotFound desc = could not find container \"b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1\": container with ID starting with b6ddd3454cb16079aa26735957d3572caa4d85a9d03a2bd43d6852de5ab380e1 not found: ID does not exist" Oct 30 09:42:10.756312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:10.756290 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6d4b7a-openshift-default-7bbb88b5dc-lbv9m"] Oct 30 09:42:11.366317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:11.366286 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 09:42:11.366500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:11.366336 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 09:42:11.572654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:11.572625 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:12.365496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:12.365459 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 09:42:12.365690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:12.365542 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 09:42:12.403243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:12.400043 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59257381-27fa-487d-b6ea-44311a26584c" path="/var/lib/kubelet/pods/59257381-27fa-487d-b6ea-44311a26584c/volumes" Oct 30 09:42:13.147260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:13.147196 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:13.370455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:13.370427 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:42:13.370771 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:13.370752 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:42:13.371444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:13.371427 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 09:42:14.731126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:14.731089 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:16.326235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:16.326192 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:17.993863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:17.993824 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:18.131041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.131004 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:18.556429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.556396 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:42:18.557183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557153 4920 cpu_manager.go:414] "containerMap: removing container" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.557183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557180 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557231 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" containerName="maas-api" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557238 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" containerName="maas-api" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557257 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8b03b18c-e6be-466a-b60a-6408e9b5d60d" containerName="revoke-keys" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557262 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b03b18c-e6be-466a-b60a-6408e9b5d60d" containerName="revoke-keys" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557271 4920 cpu_manager.go:414] "containerMap: removing container" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.557410 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557276 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.557610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557417 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.557610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557427 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f65189e-ee53-4cd2-92f3-6f1cfe5fe4cf" containerName="maas-api" Oct 30 09:42:18.557610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557436 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b03b18c-e6be-466a-b60a-6408e9b5d60d" containerName="revoke-keys" Oct 30 09:42:18.557610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.557442 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="59257381-27fa-487d-b6ea-44311a26584c" containerName="istio-proxy" Oct 30 09:42:18.562721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.562696 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.587877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.587814 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:42:18.591352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.591311 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-88cbc7-openshift-default-dockercfg-vdjv6\"" Oct 30 09:42:18.603975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.603941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:18.689950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.689918 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.689965 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.689983 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r8h\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690151 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690299 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690357 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690432 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690483 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.690580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.690517 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.700951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.700924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804275 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804351 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804379 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r8h\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804453 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804556 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804624 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804664 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804726 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.804780 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.805295 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.805568 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.815290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.805815 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.836358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.816642 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.843241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.839833 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r8h\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.856067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.853389 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.857137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.857095 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.878240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.866284 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:18.900317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:18.899917 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\") pod \"e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:19.179801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.179710 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:19.503519 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:42:19.503484 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a86afa_224a_40ea_8eba_14c2513182b1.slice/crio-c10618b69205ae5aeb7976ddfff11f36d7db514e6c334ff32f16c1f5c457ec76 WatchSource:0}: Error finding container c10618b69205ae5aeb7976ddfff11f36d7db514e6c334ff32f16c1f5c457ec76: Status 404 returned error can't find the container with id c10618b69205ae5aeb7976ddfff11f36d7db514e6c334ff32f16c1f5c457ec76 Oct 30 09:42:19.505966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.505936 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:42:19.506640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.506607 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:19.506754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.506696 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:19.506754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.506741 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:42:19.746956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.746878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:19.857329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.857285 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerStarted","Data":"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c"} Oct 30 09:42:19.857541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.857360 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerStarted","Data":"c10618b69205ae5aeb7976ddfff11f36d7db514e6c334ff32f16c1f5c457ec76"} Oct 30 09:42:19.901866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:19.901804 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podStartSLOduration=1.9017492950000001 podStartE2EDuration="1.901749295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:19.897310842 +0000 UTC m=+2903.620738006" watchObservedRunningTime="2026-10-30 09:42:19.901749295 +0000 UTC m=+2903.625176441" Oct 30 09:42:20.180580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:20.180499 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:20.186908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:20.186877 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:20.864731 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:20.864696 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:20.865707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:20.865688 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:42:21.555597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:21.555567 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:22.229468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:22.229427 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:22.327153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:22.327112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:23.264694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:23.264655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:23.940194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:23.940147 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:24.036330 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:24.036292 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:24.952941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:24.952908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:26.498986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:26.498945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:28.051074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:28.051040 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:29.655465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:29.655429 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:30.594609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:30.594574 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:30.693741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:30.693710 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:31.235242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:31.234269 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:31.237783 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:31.237705 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:42:31.546304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:31.546268 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:33.058180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:33.058130 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:34.648915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:34.648855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:36.292807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:36.292772 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:37.520268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:37.520229 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:37.612792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:37.612750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:37.704695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:37.704655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:37.956992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:37.956959 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:39.408540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:39.408455 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:40.960920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:40.960887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:42.526308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:42.526274 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:43.313108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:43.313073 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:43.419584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:43.419551 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:43.545982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:43.545943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:43.792184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:43.792130 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:43.955457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:43.955427 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:44.064553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:44.064457 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:44.191538 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:44.191507 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:44.378800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:44.378708 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:44.533919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:44.533882 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:42:44.952236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:44.952184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:45.053697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:45.053666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:46.060707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:46.060670 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:46.436252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:46.436129 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:47.875051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:47.875012 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:49.456111 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:49.456036 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:49.982366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:49.982315 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:50.082291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:50.082181 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:51.454024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:51.453982 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:52.098580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:52.098543 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:52.235171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:52.235140 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:52.944845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:52.944813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:53.491675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:53.491641 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:42:53.491904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:53.491875 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" containerID="cri-o://7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743" gracePeriod=30 Oct 30 09:42:54.428343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:54.428291 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:55.964688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:55.964646 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:57.470254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:57.470204 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:57.936053 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:57.935994 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:58.020471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.020418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:58.805654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.805625 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:42:58.861656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861610 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.861872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861674 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.861872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861741 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.861872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861851 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.862046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861893 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.862046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.861974 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.862153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.862084 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.862153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.862124 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.862279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.862239 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\") pod \"63e00b46-5f10-4079-87a8-8e063fba6cbc\" (UID: \"63e00b46-5f10-4079-87a8-8e063fba6cbc\") " Oct 30 09:42:58.864047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.863774 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:58.864402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.864261 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:58.864545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.864500 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:58.868262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.868208 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:58.868642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.868620 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:58.869434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.869199 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "kube-api-access-sbvrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:58.871292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.869647 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:42:58.871292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.870908 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:58.875366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.875262 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy" pod "63e00b46-5f10-4079-87a8-8e063fba6cbc" (UID: "63e00b46-5f10-4079-87a8-8e063fba6cbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:42:58.949540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.949467 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:42:58.964584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964542 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964576 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964589 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964600 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964612 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964622 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e00b46-5f10-4079-87a8-8e063fba6cbc-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964634 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvrw\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-kube-api-access-sbvrw\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964646 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e00b46-5f10-4079-87a8-8e063fba6cbc-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:58.964841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:58.964657 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e00b46-5f10-4079-87a8-8e063fba6cbc-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:42:59.250276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.250242 4920 generic.go:356] "Generic (PLEG): container finished" podID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerID="7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743" exitCode=0 Oct 30 09:42:59.250458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.250283 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerDied","Data":"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743"} Oct 30 09:42:59.250458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.250328 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" Oct 30 09:42:59.250458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.250388 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb" event={"ID":"63e00b46-5f10-4079-87a8-8e063fba6cbc","Type":"ContainerDied","Data":"17e7c33752bbd7b5e2b92313bdd3b28e48085105c7037846889dea0caa0bb1fc"} Oct 30 09:42:59.250458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.250413 4920 scope.go:122] "RemoveContainer" containerID="7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743" Oct 30 09:42:59.262130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.262110 4920 scope.go:122] "RemoveContainer" containerID="6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8" Oct 30 09:42:59.272394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.272372 4920 scope.go:122] "RemoveContainer" containerID="7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743" Oct 30 09:42:59.272675 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:59.272655 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743\": container with ID starting with 7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743 not found: ID does not exist" containerID="7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743" Oct 30 09:42:59.272726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.272685 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743"} err="failed to get container status \"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743\": rpc error: code = NotFound desc = could not find container \"7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743\": container with ID starting with 7ba7b7cc49a40c67e692c00073effbf9d0a62bb755eee168f84c5d1ba9df8743 not found: ID does not exist" Oct 30 09:42:59.272726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.272702 4920 scope.go:122] "RemoveContainer" containerID="6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8" Oct 30 09:42:59.272974 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:42:59.272943 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8\": container with ID starting with 6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8 not found: ID does not exist" containerID="6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8" Oct 30 09:42:59.273081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.272974 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8"} err="failed to get container status \"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8\": rpc error: code = NotFound desc = could not find container \"6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8\": container with ID starting with 6b4b7400b707d121e63639cfe413d6db02e5eed02d7209c444b3b4fca248d5e8 not found: ID does not exist" Oct 30 09:42:59.304726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.304690 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:42:59.314684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:42:59.314646 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-httjb"] Oct 30 09:43:00.398551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.398517 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" path="/var/lib/kubelet/pods/63e00b46-5f10-4079-87a8-8e063fba6cbc/volumes" Oct 30 09:43:00.470379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.470345 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:43:00.471005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.470990 4920 cpu_manager.go:414] "containerMap: removing container" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:00.471078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.471006 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:00.471078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.471026 4920 cpu_manager.go:414] "containerMap: removing container" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:00.471078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.471031 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:00.471228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.471126 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:00.476337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.476318 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.506843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.506808 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:43:00.507707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.507678 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-88cbc7-serving-cert\"" Oct 30 09:43:00.535886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.535852 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:00.605131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.605092 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.605346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.605315 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.605402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.605344 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5md\" (UniqueName: \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.605590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.605562 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.644391 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.644333 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:00.707271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.707172 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.707455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.707329 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.707536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.707452 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.707536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.707473 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5md\" (UniqueName: \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.708193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.708162 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.709862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.709836 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.710012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.709988 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.757185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.756777 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5md\" (UniqueName: \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\") pod \"maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:00.767845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.767799 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:00.802893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:00.802857 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:01.112081 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:43:01.112036 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ce10d9_d408_497c_9b17_42ec3433d78a.slice/crio-a712d1dbcad4c29dc3dbda5ffd16698125c293d1f29dfe8204beffe2bc16447f WatchSource:0}: Error finding container a712d1dbcad4c29dc3dbda5ffd16698125c293d1f29dfe8204beffe2bc16447f: Status 404 returned error can't find the container with id a712d1dbcad4c29dc3dbda5ffd16698125c293d1f29dfe8204beffe2bc16447f Oct 30 09:43:01.113826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:01.113806 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:43:01.115997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:01.115971 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:43:01.141963 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:43:01.141925 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:43:01.266914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:01.266882 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" event={"ID":"66ce10d9-d408-497c-9b17-42ec3433d78a","Type":"ContainerStarted","Data":"a712d1dbcad4c29dc3dbda5ffd16698125c293d1f29dfe8204beffe2bc16447f"} Oct 30 09:43:02.228324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:02.228286 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:02.275948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:02.275914 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" event={"ID":"66ce10d9-d408-497c-9b17-42ec3433d78a","Type":"ContainerStarted","Data":"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b"} Oct 30 09:43:02.276134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:02.276008 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:02.327239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:02.326476 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" podStartSLOduration=1.8299043259999999 podStartE2EDuration="2.326455678s" totalImagesPullingTime="496.551352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:02.325387013 +0000 UTC m=+2946.048814194" watchObservedRunningTime="2026-10-30 09:43:02.326455678 +0000 UTC m=+2946.049882823" Oct 30 09:43:03.786557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:03.786510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:05.389393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:05.389352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:06.684794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:06.684748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:07.148983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:07.148952 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:08.290954 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:08.290920 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:43:08.691280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:08.691187 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:09.639719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:09.639679 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:09.738787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:09.738743 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:10.704783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:10.704749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:11.987511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:11.987479 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:12.392852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:12.392819 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:13.185033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:13.185000 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:13.493276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:13.493175 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:13.625812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:13.625753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:14.334826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:14.334785 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:14.749709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:14.749620 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:14.885869 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:14.885836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:16.286170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:16.286137 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:17.080516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:17.080480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:17.185774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:17.185698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:18.223044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:18.223003 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:18.511693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:18.511598 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:19.236963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:19.236932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:19.522798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:19.522703 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:19.620090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:19.620057 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:20.060517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:20.060478 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:20.396821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:20.396713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:20.516265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:20.505809 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:21.637289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:21.637243 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:23.349684 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:23.349645 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:24.917518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:24.917455 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:26.463547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:26.463504 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:27.948418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:27.948387 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:29.408061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:29.408024 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:30.963929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:30.963897 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:31.137567 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:43:31.137527 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:43:32.476127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:32.476090 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:33.111297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:33.111260 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:33.208924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:33.208884 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:33.758398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:33.758365 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:34.320984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:34.320934 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:35.813605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:35.813570 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:37.465903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:37.465862 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:38.696483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:38.696436 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:43:39.169301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:39.169270 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:40.297207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:40.297152 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:40.435315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:40.435278 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:40.557125 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:40.557034 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:40.913445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:40.913347 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:42.078378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:42.078342 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:42.178842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:42.178803 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:42.771990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:42.771950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:44.569428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:44.569378 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:45.531811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:45.531765 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:45.629158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:45.629123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:46.646877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:46.646749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:46.783238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:46.768974 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:46.890956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:46.890915 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:48.526136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:48.526053 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:48.627247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:48.627196 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:48.724251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:48.724197 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:49.240049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:49.240015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:50.477822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:50.477769 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:50.640180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:50.640139 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:50.939518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:50.938443 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": read tcp 10.132.0.2:38540->10.132.0.250:15021: read: connection reset by peer" start-of-body= Oct 30 09:43:50.939518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:50.938504 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": read tcp 10.132.0.2:38540->10.132.0.250:15021: read: connection reset by peer" Oct 30 09:43:51.070046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.070012 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:43:51.071030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.071005 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e00b46-5f10-4079-87a8-8e063fba6cbc" containerName="istio-proxy" Oct 30 09:43:51.074800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.074777 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.093244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.093177 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d66d22-openshift-default-dockercfg-zf9tn\"" Oct 30 09:43:51.107878 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.107839 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:43:51.132647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.132587 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:51.171478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171439 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171485 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171609 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171648 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wft\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171814 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171870 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171900 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.171981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.171951 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.172045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.172031 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.274587 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.274654 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.274693 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.274797 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.274840 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wft\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.275016 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.275087 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.275114 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.275146 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.275735 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.280413 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.293566 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.294831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.293834 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.296303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.296125 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.319239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.310560 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.332242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.332140 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.353264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.339009 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wft\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.364405 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.364338 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\") pod \"e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.406243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.396440 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:51.817969 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:43:51.817930 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f56fec_70e4_4d0c_80e6_4bf5f4b3f32f.slice/crio-faa78504a7a305c60175db3756bd19aae3a2dd5cc512f9ba0a1943349546ece2 WatchSource:0}: Error finding container faa78504a7a305c60175db3756bd19aae3a2dd5cc512f9ba0a1943349546ece2: Status 404 returned error can't find the container with id faa78504a7a305c60175db3756bd19aae3a2dd5cc512f9ba0a1943349546ece2 Oct 30 09:43:51.817969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.817950 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:43:51.820925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.820890 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.821052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.820976 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.821052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.821005 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.934099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.934062 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerStarted","Data":"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083"} Oct 30 09:43:51.934317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.934131 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerStarted","Data":"faa78504a7a305c60175db3756bd19aae3a2dd5cc512f9ba0a1943349546ece2"} Oct 30 09:43:51.936129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.936102 4920 generic.go:356] "Generic (PLEG): container finished" podID="61a86afa-224a-40ea-8eba-14c2513182b1" containerID="7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c" exitCode=0 Oct 30 09:43:51.936279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.936173 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerDied","Data":"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c"} Oct 30 09:43:51.936707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.936686 4920 scope.go:122] "RemoveContainer" containerID="7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c" Oct 30 09:43:51.939048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.939005 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.939164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.939098 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.939164 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.939147 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:43:51.969751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.969718 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:43:51.970070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.970001 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" containerID="cri-o://2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d" gracePeriod=30 Oct 30 09:43:51.985425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:51.985371 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podStartSLOduration=0.985354489 podStartE2EDuration="985.354489ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:51.980731987 +0000 UTC m=+2995.704159159" watchObservedRunningTime="2026-10-30 09:43:51.985354489 +0000 UTC m=+2995.708781632" Oct 30 09:43:52.398789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:52.398689 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:43:52.398789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:52.398762 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:43:52.403324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:52.403213 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:52.839952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:52.839913 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:52.947516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:52.947476 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerStarted","Data":"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23"} Oct 30 09:43:53.180381 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.180296 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:43:53.180555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.180511 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:53.180633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.180566 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:53.451300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.446577 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:53.955429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.955394 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:53.956463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:53.956441 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:43:54.180946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:54.180909 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:54.181129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:54.180966 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:54.255789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:54.255673 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:54.603246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:54.586139 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:54.695240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:54.694836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:55.180504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:55.180473 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:55.180920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:55.180525 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:56.106244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:56.097615 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:56.182382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:56.182342 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:56.182890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:56.182486 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:56.470255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:56.470136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:56.566910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:56.566870 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:57.180680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.180646 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:57.180853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.180706 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:57.763352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.763322 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:43:57.871642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.871558 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.871851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.871669 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.871851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.871762 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.871851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.871835 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.872043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.871882 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.872043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872005 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.872190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872123 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.872190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872138 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:43:57.872642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872618 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:43:57.872798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872660 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:57.872955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.872736 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:43:57.873077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.873054 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.873335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.873310 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\") pod \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\" (UID: \"8428abf5-dfd4-4a0f-9290-ddce75bb2735\") " Oct 30 09:43:57.873682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.873661 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:43:57.874204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874172 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.874386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874370 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.874499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874487 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.874600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874588 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.874744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874687 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.874842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874816 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:43:57.874943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.874919 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:43:57.875205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.875182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:57.876315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.876290 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh" pod "8428abf5-dfd4-4a0f-9290-ddce75bb2735" (UID: "8428abf5-dfd4-4a0f-9290-ddce75bb2735"). InnerVolumeSpecName "kube-api-access-blfwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:57.976161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.976125 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfwh\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-kube-api-access-blfwh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.976161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.976156 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.976161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.976168 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:57.976460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:57.976176 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8428abf5-dfd4-4a0f-9290-ddce75bb2735-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:43:58.047237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.041539 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:43:58.078780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.078743 4920 generic.go:356] "Generic (PLEG): container finished" podID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerID="2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d" exitCode=0 Oct 30 09:43:58.078960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.078797 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" Oct 30 09:43:58.078960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.078808 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" event={"ID":"8428abf5-dfd4-4a0f-9290-ddce75bb2735","Type":"ContainerDied","Data":"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d"} Oct 30 09:43:58.078960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.078862 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm" event={"ID":"8428abf5-dfd4-4a0f-9290-ddce75bb2735","Type":"ContainerDied","Data":"afb6a8c78e4cfc74329753965e8a13bd4d6b92639112c79829a62b1a8397eb15"} Oct 30 09:43:58.078960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.078894 4920 scope.go:122] "RemoveContainer" containerID="2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d" Oct 30 09:43:58.093373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.093351 4920 scope.go:122] "RemoveContainer" containerID="2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d" Oct 30 09:43:58.093708 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:43:58.093690 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d\": container with ID starting with 2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d not found: ID does not exist" containerID="2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d" Oct 30 09:43:58.093750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.093721 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d"} err="failed to get container status \"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d\": rpc error: code = NotFound desc = could not find container \"2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d\": container with ID starting with 2af17f28aa79c085d17ba9ef53fecc073d6ec90ee8540f6b5f23b57d26105f4d not found: ID does not exist" Oct 30 09:43:58.142245 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.142138 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:43:58.146162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.146135 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dszbm"] Oct 30 09:43:58.180386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.180353 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 30 09:43:58.180571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.180404 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 30 09:43:58.422403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:58.419871 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" path="/var/lib/kubelet/pods/8428abf5-dfd4-4a0f-9290-ddce75bb2735/volumes" Oct 30 09:43:59.185583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:59.185549 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:43:59.186062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:59.186023 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:43:59.186781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:59.186755 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:43:59.654579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:43:59.654522 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:01.137206 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:44:01.137171 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:44:01.240023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:01.239971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:02.876527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:02.876489 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:03.371876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:03.371844 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:44:03.465264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:03.465206 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:44:04.591326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:04.591294 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:06.185448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:06.185414 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:07.778739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:07.778677 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:09.483379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:09.483325 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:10.420048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.420011 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:44:10.420712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.420694 4920 cpu_manager.go:414] "containerMap: removing container" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" Oct 30 09:44:10.420770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.420714 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" Oct 30 09:44:10.420859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.420849 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="8428abf5-dfd4-4a0f-9290-ddce75bb2735" containerName="istio-proxy" Oct 30 09:44:10.426231 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.426201 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.459812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.459778 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:44:10.493098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493064 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493198 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493270 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493312 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493352 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493407 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493437 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5mz\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493470 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.493976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.493602 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595276 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595378 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595440 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595457 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595473 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595498 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595538 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595570 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5mz\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595602 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.595986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.595855 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.596201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.596171 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.596320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.596230 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.596320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.596259 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.596320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.596268 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.597974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.597953 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.598165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.598151 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.614930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.614893 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.615171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.615143 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5mz\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\") pod \"maas-default-gateway-openshift-default-77f556d647-knmrm\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:10.752943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:10.752855 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:11.072304 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:44:11.072269 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3ca2c6_df41_474b_981e_388470fc0bea.slice/crio-fd6bca17c2eaf843cc9270495aeeea99bd1d7b5ca8ff706ce65e4da37d804d48 WatchSource:0}: Error finding container fd6bca17c2eaf843cc9270495aeeea99bd1d7b5ca8ff706ce65e4da37d804d48: Status 404 returned error can't find the container with id fd6bca17c2eaf843cc9270495aeeea99bd1d7b5ca8ff706ce65e4da37d804d48 Oct 30 09:44:11.073885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.073856 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:44:11.075521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.075489 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:11.075608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.075553 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:11.075608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.075596 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:11.187821 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.187782 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" event={"ID":"ab3ca2c6-df41-474b-981e-388470fc0bea","Type":"ContainerStarted","Data":"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10"} Oct 30 09:44:11.188021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.187849 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" event={"ID":"ab3ca2c6-df41-474b-981e-388470fc0bea","Type":"ContainerStarted","Data":"fd6bca17c2eaf843cc9270495aeeea99bd1d7b5ca8ff706ce65e4da37d804d48"} Oct 30 09:44:11.195607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.195575 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:11.271498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.271435 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podStartSLOduration=1.2714113710000001 podStartE2EDuration="1.271411371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:11.229584438 +0000 UTC m=+3014.953011626" watchObservedRunningTime="2026-10-30 09:44:11.271411371 +0000 UTC m=+3014.994838515" Oct 30 09:44:11.753051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:11.753014 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:12.755959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:12.755920 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:12.756430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:12.755987 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:12.987450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:12.987411 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:13.753501 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:13.753464 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:13.753686 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:13.753524 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:14.531614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:14.531558 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:14.753882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:14.753846 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:14.754074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:14.753902 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:15.753520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:15.753484 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:15.753927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:15.753541 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:16.167326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:16.167289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:16.753450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:16.753415 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:16.753689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:16.753474 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:17.754173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:17.754133 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:17.754628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:17.754185 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:18.006422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:18.006298 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:18.753747 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:18.753703 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 30 09:44:18.753936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:18.753768 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 30 09:44:19.648243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:19.647561 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:19.879935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:19.879903 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:19.880632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:19.880439 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:19.881959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:19.881937 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:44:21.522612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:21.522580 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:23.401268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:23.401019 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:25.109159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.109120 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:25.471953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.471874 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:44:25.477504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.477480 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497156 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497232 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497314 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497366 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497344 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497392 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.497780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.497762 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.499374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.499353 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.499584 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.499568 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2s6s\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.515365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.515165 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:44:25.601320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601283 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601325 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601357 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601454 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601582 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601616 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601652 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2s6s\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601691 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601736 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.601880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.601801 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.602302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.602281 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.602483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.602446 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.602626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.602452 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.602626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.602600 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.604707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.604684 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.604895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.604874 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.630765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.630721 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.630924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.630854 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2s6s\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\") pod \"maas-default-gateway-openshift-default-77f556d647-tjlgk\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:25.813678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:25.813637 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:26.135336 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:44:26.135304 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a44536_43a9_4d18_986a_baa4cbff5cff.slice/crio-e0a069a43e26011187922cbb3c7b86f0b7d49f8e04e9dd1ca5422f5291c85ba5 WatchSource:0}: Error finding container e0a069a43e26011187922cbb3c7b86f0b7d49f8e04e9dd1ca5422f5291c85ba5: Status 404 returned error can't find the container with id e0a069a43e26011187922cbb3c7b86f0b7d49f8e04e9dd1ca5422f5291c85ba5 Oct 30 09:44:26.138775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.138741 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:26.138872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.138809 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:26.138872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.138836 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:26.139457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.139405 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:44:26.382152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.382060 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" event={"ID":"23a44536-43a9-4d18-986a-baa4cbff5cff","Type":"ContainerStarted","Data":"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93"} Oct 30 09:44:26.382152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.382129 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" event={"ID":"23a44536-43a9-4d18-986a-baa4cbff5cff","Type":"ContainerStarted","Data":"e0a069a43e26011187922cbb3c7b86f0b7d49f8e04e9dd1ca5422f5291c85ba5"} Oct 30 09:44:26.814638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.814604 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:26.902956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:26.902921 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:27.829242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:27.828361 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:27.829242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:27.828442 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:28.318801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:28.318762 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:28.441448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:28.441413 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:28.821716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:28.821675 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:28.822059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:28.822026 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:28.874643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:28.874607 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:29.003238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:29.002761 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:29.823254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:29.814977 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:29.823254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:29.815075 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:29.833244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:29.829847 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:29.947236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:29.947180 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:30.819374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:30.819337 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:30.819665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:30.819636 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:31.156566 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:44:31.156483 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:44:31.256419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:31.256383 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:31.820673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:31.820634 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:31.820982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:31.820949 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:32.814515 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:32.814478 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:32.815027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:32.814541 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:32.900073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:32.900032 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:33.680286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:33.680196 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podStartSLOduration=8.680178974 podStartE2EDuration="8.680178974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:26.441146313 +0000 UTC m=+3030.164573484" watchObservedRunningTime="2026-10-30 09:44:33.680178974 +0000 UTC m=+3037.403606117" Oct 30 09:44:33.681734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:33.681705 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:44:33.682165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:33.682132 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" containerID="cri-o://381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af" gracePeriod=30 Oct 30 09:44:33.814719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:33.814668 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tjlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:44:33.815248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:33.814764 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 09:44:34.602964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:34.602910 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:35.430534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:35.430402 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:35.432007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:35.431980 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:35.433675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:35.433655 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:44:35.653471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:35.653438 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:35.770325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:35.763838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:36.522240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:36.521977 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:38.426523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:38.426451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:38.609241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:38.601638 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:38.970019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:38.969936 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 09:44:38.970019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:38.969996 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 09:44:39.408804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.408776 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:44:39.574179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574086 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574144 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574182 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574243 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574323 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574402 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574558 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574638 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574683 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:39.574938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574707 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\") pod \"2947718f-2e49-48c9-abf1-0a224f149c61\" (UID: \"2947718f-2e49-48c9-abf1-0a224f149c61\") " Oct 30 09:44:39.575281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574707 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:39.575281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.574936 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:39.575281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575045 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:39.575412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575350 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.575412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575374 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2947718f-2e49-48c9-abf1-0a224f149c61-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.575412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575387 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.575412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575399 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.575626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.575457 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:39.576931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.576906 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:39.577046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.576945 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "kube-api-access-ksvm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:39.577046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.576986 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:44:39.577133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.577123 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo" pod "2947718f-2e49-48c9-abf1-0a224f149c61" (UID: "2947718f-2e49-48c9-abf1-0a224f149c61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.701560 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2947718f-2e49-48c9-abf1-0a224f149c61-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.701603 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.701619 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvm6\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-kube-api-access-ksvm6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.701634 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2947718f-2e49-48c9-abf1-0a224f149c61-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.701647 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2947718f-2e49-48c9-abf1-0a224f149c61-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:44:39.704293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.704170 4920 generic.go:356] "Generic (PLEG): container finished" podID="2947718f-2e49-48c9-abf1-0a224f149c61" containerID="381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af" exitCode=0 Oct 30 09:44:39.704804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.704503 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerDied","Data":"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af"} Oct 30 09:44:39.704804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.704597 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" event={"ID":"2947718f-2e49-48c9-abf1-0a224f149c61","Type":"ContainerDied","Data":"02d53aeabb1d5c62d201de837fbd9c353da2f51a25c4a568e69d40266eeadaaf"} Oct 30 09:44:39.704804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.704629 4920 scope.go:122] "RemoveContainer" containerID="381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af" Oct 30 09:44:39.704804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.704801 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn" Oct 30 09:44:39.743411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.743370 4920 scope.go:122] "RemoveContainer" containerID="d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29" Oct 30 09:44:39.755465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.755400 4920 scope.go:122] "RemoveContainer" containerID="381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af" Oct 30 09:44:39.755876 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:44:39.755845 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af\": container with ID starting with 381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af not found: ID does not exist" containerID="381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af" Oct 30 09:44:39.755964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.755888 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af"} err="failed to get container status \"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af\": rpc error: code = NotFound desc = could not find container \"381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af\": container with ID starting with 381a4f3605bb19cd2418257c295dbce57c98b365cb24eedfe49b9ebd050729af not found: ID does not exist" Oct 30 09:44:39.755964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.755912 4920 scope.go:122] "RemoveContainer" containerID="d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29" Oct 30 09:44:39.756320 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:44:39.756298 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29\": container with ID starting with d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29 not found: ID does not exist" containerID="d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29" Oct 30 09:44:39.756426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.756325 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29"} err="failed to get container status \"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29\": rpc error: code = NotFound desc = could not find container \"d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29\": container with ID starting with d6d82dfe43d68f7e1bfcb489813adb5a2c17aee59c19336c7426cfe463eeaa29 not found: ID does not exist" Oct 30 09:44:39.776631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.776597 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:44:39.788754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:39.788725 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-hclgn"] Oct 30 09:44:40.311650 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:40.311611 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:40.401032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:40.401001 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" path="/var/lib/kubelet/pods/2947718f-2e49-48c9-abf1-0a224f149c61/volumes" Oct 30 09:44:41.968357 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:41.968323 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:43.595187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:43.595153 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:45.241013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:45.240969 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:46.946342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:46.946308 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:47.595633 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:47.595597 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:47.743250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:47.741939 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:48.897250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:48.897189 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:50.436907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:50.436871 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:52.096537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:52.096496 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:52.404390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:52.404273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:53.660187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:53.660152 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:55.147029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:55.146988 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:56.777907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:56.777874 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:57.763818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:57.763769 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:44:58.407743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:58.407712 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:59.671604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:59.671568 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:44:59.768184 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:44:59.768127 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:00.201070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.201034 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:00.215248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.202394 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.215502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.215476 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.215669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.215655 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.215776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.215762 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.216122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.216101 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.216288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.216273 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2947718f-2e49-48c9-abf1-0a224f149c61" containerName="istio-proxy" Oct 30 09:45:00.258421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.258387 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:00.268265 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.268236 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.312711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.308021 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.349236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.349189 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:00.355377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.355333 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.396369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.396334 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:00.443630 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.398243 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5r9\" (UniqueName: \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.443966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.443937 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.444165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.444147 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnjm\" (UniqueName: \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.444499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.444477 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.444864 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.399194 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:00.445096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.399419 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:45:00.491229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.486999 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:00.537524 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.537492 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:00.539572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.539551 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.545754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.545721 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.545905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.545780 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.545977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.545927 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5r9\" (UniqueName: \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.545977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.545964 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.546103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.546006 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.546103 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.546083 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnjm\" (UniqueName: \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.547324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.547264 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.547585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.547540 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.548991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.548891 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:00.557280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.557256 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:00.557468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.557452 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:00.557601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.557587 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:00.558948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.558923 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.564732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.564711 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:00.568179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.568158 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.571692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.571512 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:00.571692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.571582 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.602242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.584079 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnjm\" (UniqueName: \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.611556 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.610298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5r9\" (UniqueName: \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.638303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.629724 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:00.638729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.638693 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655041 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6wl\" (UniqueName: \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655108 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655212 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655327 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lvx\" (UniqueName: \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655371 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5p77\" (UniqueName: \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655433 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655540 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655576 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655603 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsllh\" (UniqueName: \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.655656 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.656994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.656930 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.672529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.672499 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:00.673071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.673051 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.695645 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.695728 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.695758 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.695797 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.696637 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.740249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.704584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.759703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.759618 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lvx\" (UniqueName: \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.759882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.759734 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5p77\" (UniqueName: \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.759882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.759793 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.759882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.759837 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.760077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.759956 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.760077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760005 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsllh\" (UniqueName: \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.760303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760275 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.760418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760397 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.760527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760509 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6wl\" (UniqueName: \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.760600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760570 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.760662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760645 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94485\" (UniqueName: \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.760736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.760720 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7l6\" (UniqueName: \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.763742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.763709 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.764918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.764890 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.765612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.765584 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.766324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.766298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.780207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.780174 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6wl\" (UniqueName: \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.780786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.780758 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lvx\" (UniqueName: \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:00.780882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.780755 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsllh\" (UniqueName: \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\") pod \"maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:00.782019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.781995 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5p77\" (UniqueName: \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:00.784826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.784803 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:00.858927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.858893 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:00.861886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.861841 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.862023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.861990 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94485\" (UniqueName: \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.862067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.862049 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7l6\" (UniqueName: \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.862763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.862190 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.862763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.862701 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.864837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.863761 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.864837 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.864067 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:00.874084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.874060 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:00.881594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.881565 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94485\" (UniqueName: \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:00.884187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.881929 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7l6\" (UniqueName: \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\") pod \"maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:00.974894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:00.974432 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:01.064171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.063949 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:01.086002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.085707 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:01.169932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.169841 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:01.170071 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:45:01.170030 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a865623_785c_4e1b_a6df_3748ff6d140f.slice/crio-c25da19e071b74625da051120fa045e263e91d228a2fd8c05419cdc18eee8fea WatchSource:0}: Error finding container c25da19e071b74625da051120fa045e263e91d228a2fd8c05419cdc18eee8fea: Status 404 returned error can't find the container with id c25da19e071b74625da051120fa045e263e91d228a2fd8c05419cdc18eee8fea Oct 30 09:45:01.173599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.173563 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:01.185264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.185111 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:01.220113 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:01.220008 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:45:01.547346 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.546770 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:01.558007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.557081 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:01.584565 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:45:01.584460 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622f3ba5_37a8_4777_b470_ca325878fbcf.slice/crio-a0fa9d5cd227a10f99e3a6c20bf619de40ddab4475860e61818218f149c23b42 WatchSource:0}: Error finding container a0fa9d5cd227a10f99e3a6c20bf619de40ddab4475860e61818218f149c23b42: Status 404 returned error can't find the container with id a0fa9d5cd227a10f99e3a6c20bf619de40ddab4475860e61818218f149c23b42 Oct 30 09:45:01.586889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.586729 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:01.850867 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.850838 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:01.935267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.935234 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:01.943399 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:45:01.943369 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49320e8c_dd17_474a_b5c0_8d2cec590723.slice/crio-9669985a0c104531286b27daa4931ce376e4c54855621dc2d8011d52df3b751d WatchSource:0}: Error finding container 9669985a0c104531286b27daa4931ce376e4c54855621dc2d8011d52df3b751d: Status 404 returned error can't find the container with id 9669985a0c104531286b27daa4931ce376e4c54855621dc2d8011d52df3b751d Oct 30 09:45:01.947019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.946991 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:01.980513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.980467 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:01.985906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:01.984837 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:01.988384 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:45:01.988355 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efb2d2e_2f2b_4b2f_990e_83a91e0165f3.slice/crio-15cee9e57336382538b649cb433e3c7922d2bc0fdeb439b0fac90509907b153b WatchSource:0}: Error finding container 15cee9e57336382538b649cb433e3c7922d2bc0fdeb439b0fac90509907b153b: Status 404 returned error can't find the container with id 15cee9e57336382538b649cb433e3c7922d2bc0fdeb439b0fac90509907b153b Oct 30 09:45:02.039967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.039916 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerStarted","Data":"30a6ac3b8dd4fb1bba45ac27fd343e9be1f3b9899ba5182499e4a4846c66baba"} Oct 30 09:45:02.041606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.041582 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerStarted","Data":"67d5ee120d15b8e66fba1130860d8d45ffecaca8d8537816c3e06447c4746d89"} Oct 30 09:45:02.042910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.042882 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerStarted","Data":"8f55c3975da830560a1b1600cce3a71972188c9c39e8e164baf80af6f21fda76"} Oct 30 09:45:02.044553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.044382 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerStarted","Data":"5f68ee4d8b3e8e0f80fed36aa2f40bb82bc6aa0ce974695e59bdf4d0cf5a295d"} Oct 30 09:45:02.045940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.045920 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerStarted","Data":"5d411b021fce750895752201a0ce886935cf4535f83795327e6e018ed2219c01"} Oct 30 09:45:02.047146 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.047126 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerStarted","Data":"a0fa9d5cd227a10f99e3a6c20bf619de40ddab4475860e61818218f149c23b42"} Oct 30 09:45:02.048363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.048344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerStarted","Data":"9669985a0c104531286b27daa4931ce376e4c54855621dc2d8011d52df3b751d"} Oct 30 09:45:02.049720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.049698 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerStarted","Data":"892c9f19239b132199e3ac4be374cf41996de0d57068124506ab967aa29f1cd1"} Oct 30 09:45:02.049845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.049734 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerStarted","Data":"c25da19e071b74625da051120fa045e263e91d228a2fd8c05419cdc18eee8fea"} Oct 30 09:45:02.091774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.091663 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" podStartSLOduration=2.091648055 podStartE2EDuration="2.091648055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.087460161 +0000 UTC m=+3065.810887300" watchObservedRunningTime="2026-10-30 09:45:02.091648055 +0000 UTC m=+3065.815075200" Oct 30 09:45:02.269284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:02.269243 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:03.060536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.060507 4920 generic.go:356] "Generic (PLEG): container finished" podID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerID="e5e7e8658c529a2a97ae4c345bcb54034069829997e6d88030d6b09d390c18f0" exitCode=60 Oct 30 09:45:03.060980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.060581 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerDied","Data":"e5e7e8658c529a2a97ae4c345bcb54034069829997e6d88030d6b09d390c18f0"} Oct 30 09:45:03.061061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.061051 4920 scope.go:122] "RemoveContainer" containerID="e5e7e8658c529a2a97ae4c345bcb54034069829997e6d88030d6b09d390c18f0" Oct 30 09:45:03.063005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.062986 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerStarted","Data":"320cf161ca9bec6149d8fd92d0b41b10a0376337dcb634fb2f5b6d788ac223ca"} Oct 30 09:45:03.063097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.063026 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerStarted","Data":"15cee9e57336382538b649cb433e3c7922d2bc0fdeb439b0fac90509907b153b"} Oct 30 09:45:03.064980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.064958 4920 generic.go:356] "Generic (PLEG): container finished" podID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerID="7935a6fa049c8974ad71879241a0cfb0dcae5897bb9be16b0bc9c5b5e38bb110" exitCode=60 Oct 30 09:45:03.065068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.065052 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerDied","Data":"7935a6fa049c8974ad71879241a0cfb0dcae5897bb9be16b0bc9c5b5e38bb110"} Oct 30 09:45:03.065469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.065450 4920 scope.go:122] "RemoveContainer" containerID="7935a6fa049c8974ad71879241a0cfb0dcae5897bb9be16b0bc9c5b5e38bb110" Oct 30 09:45:03.070281 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.067918 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerStarted","Data":"6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07"} Oct 30 09:45:03.076387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.075943 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerStarted","Data":"a0c5fb28d7b78171329912b1c374da55d0591d2645dc6d8e39b000a71c1d08a8"} Oct 30 09:45:03.078663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.078301 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerStarted","Data":"c3f947f8ab339c78df7f7c70bbc6c7d0f886daee086b27d40424add7b26914da"} Oct 30 09:45:03.080923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.080617 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerStarted","Data":"5340bce152abccf9935016b3a47a6e684c92cc0a43cba8096be52c54273e58d1"} Oct 30 09:45:03.083422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.082741 4920 generic.go:356] "Generic (PLEG): container finished" podID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerID="892c9f19239b132199e3ac4be374cf41996de0d57068124506ab967aa29f1cd1" exitCode=60 Oct 30 09:45:03.083422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.082866 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerDied","Data":"892c9f19239b132199e3ac4be374cf41996de0d57068124506ab967aa29f1cd1"} Oct 30 09:45:03.083588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.083568 4920 scope.go:122] "RemoveContainer" containerID="892c9f19239b132199e3ac4be374cf41996de0d57068124506ab967aa29f1cd1" Oct 30 09:45:03.086552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.086277 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerStarted","Data":"355f24f97a703c346ca041fe90de464b254f5262b0f367a281e33fe812e959f1"} Oct 30 09:45:03.194358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.194309 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" podStartSLOduration=3.194291644 podStartE2EDuration="3.194291644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.183053129 +0000 UTC m=+3066.906480275" watchObservedRunningTime="2026-10-30 09:45:03.194291644 +0000 UTC m=+3066.917718786" Oct 30 09:45:03.245098 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:03.245045 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aa4f39_3541_40e5_8209_4863af3c0a73.slice/crio-6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8aa4f39_3541_40e5_8209_4863af3c0a73.slice/crio-conmon-6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:45:03.316052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.314162 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" podStartSLOduration=3.31414197 podStartE2EDuration="3.31414197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.301628472 +0000 UTC m=+3067.025055617" watchObservedRunningTime="2026-10-30 09:45:03.31414197 +0000 UTC m=+3067.037569114" Oct 30 09:45:03.350371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.350187 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" podStartSLOduration=3.3501747809999998 podStartE2EDuration="3.350174781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.34836004 +0000 UTC m=+3067.071787182" watchObservedRunningTime="2026-10-30 09:45:03.350174781 +0000 UTC m=+3067.073601924" Oct 30 09:45:03.393795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.393747 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" podStartSLOduration=3.393733658 podStartE2EDuration="3.393733658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.389403283 +0000 UTC m=+3067.112830426" watchObservedRunningTime="2026-10-30 09:45:03.393733658 +0000 UTC m=+3067.117160819" Oct 30 09:45:03.447075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.446997 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" podStartSLOduration=3.446980677 podStartE2EDuration="3.446980677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.445472133 +0000 UTC m=+3067.168899277" watchObservedRunningTime="2026-10-30 09:45:03.446980677 +0000 UTC m=+3067.170407820" Oct 30 09:45:03.485521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.485461 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" podStartSLOduration=3.485446074 podStartE2EDuration="3.485446074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.480970942 +0000 UTC m=+3067.204398087" watchObservedRunningTime="2026-10-30 09:45:03.485446074 +0000 UTC m=+3067.208873217" Oct 30 09:45:03.853601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:03.853572 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:04.096679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.096647 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerStarted","Data":"8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36"} Oct 30 09:45:04.118819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.118786 4920 generic.go:356] "Generic (PLEG): container finished" podID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerID="355f24f97a703c346ca041fe90de464b254f5262b0f367a281e33fe812e959f1" exitCode=60 Oct 30 09:45:04.119249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.119207 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerDied","Data":"355f24f97a703c346ca041fe90de464b254f5262b0f367a281e33fe812e959f1"} Oct 30 09:45:04.120042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.120022 4920 scope.go:122] "RemoveContainer" containerID="355f24f97a703c346ca041fe90de464b254f5262b0f367a281e33fe812e959f1" Oct 30 09:45:04.136651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.136439 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerStarted","Data":"b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358"} Oct 30 09:45:04.146052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.146017 4920 generic.go:356] "Generic (PLEG): container finished" podID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerID="320cf161ca9bec6149d8fd92d0b41b10a0376337dcb634fb2f5b6d788ac223ca" exitCode=60 Oct 30 09:45:04.146204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.146156 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerDied","Data":"320cf161ca9bec6149d8fd92d0b41b10a0376337dcb634fb2f5b6d788ac223ca"} Oct 30 09:45:04.146605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.146582 4920 scope.go:122] "RemoveContainer" containerID="320cf161ca9bec6149d8fd92d0b41b10a0376337dcb634fb2f5b6d788ac223ca" Oct 30 09:45:04.152276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.151424 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerStarted","Data":"d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83"} Oct 30 09:45:04.153438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.153420 4920 generic.go:356] "Generic (PLEG): container finished" podID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerID="6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07" exitCode=60 Oct 30 09:45:04.153638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.153499 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerDied","Data":"6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07"} Oct 30 09:45:04.155804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.155784 4920 generic.go:356] "Generic (PLEG): container finished" podID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerID="a0c5fb28d7b78171329912b1c374da55d0591d2645dc6d8e39b000a71c1d08a8" exitCode=60 Oct 30 09:45:04.155984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.155882 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerDied","Data":"a0c5fb28d7b78171329912b1c374da55d0591d2645dc6d8e39b000a71c1d08a8"} Oct 30 09:45:04.156276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.156114 4920 scope.go:122] "RemoveContainer" containerID="a0c5fb28d7b78171329912b1c374da55d0591d2645dc6d8e39b000a71c1d08a8" Oct 30 09:45:04.156631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.156590 4920 scope.go:122] "RemoveContainer" containerID="6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07" Oct 30 09:45:04.157933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.157865 4920 generic.go:356] "Generic (PLEG): container finished" podID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerID="c3f947f8ab339c78df7f7c70bbc6c7d0f886daee086b27d40424add7b26914da" exitCode=60 Oct 30 09:45:04.158344 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.158319 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerDied","Data":"c3f947f8ab339c78df7f7c70bbc6c7d0f886daee086b27d40424add7b26914da"} Oct 30 09:45:04.159588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.159555 4920 scope.go:122] "RemoveContainer" containerID="c3f947f8ab339c78df7f7c70bbc6c7d0f886daee086b27d40424add7b26914da" Oct 30 09:45:04.160024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.160003 4920 generic.go:356] "Generic (PLEG): container finished" podID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerID="5340bce152abccf9935016b3a47a6e684c92cc0a43cba8096be52c54273e58d1" exitCode=60 Oct 30 09:45:04.160109 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.160053 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerDied","Data":"5340bce152abccf9935016b3a47a6e684c92cc0a43cba8096be52c54273e58d1"} Oct 30 09:45:04.160395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.160377 4920 scope.go:122] "RemoveContainer" containerID="5340bce152abccf9935016b3a47a6e684c92cc0a43cba8096be52c54273e58d1" Oct 30 09:45:04.179248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.179188 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" podStartSLOduration=4.179173148 podStartE2EDuration="4.179173148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:04.177441996 +0000 UTC m=+3067.900869158" watchObservedRunningTime="2026-10-30 09:45:04.179173148 +0000 UTC m=+3067.902600290" Oct 30 09:45:04.384447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:04.384335 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" podStartSLOduration=4.384321222 podStartE2EDuration="4.384321222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:04.38265555 +0000 UTC m=+3068.106082693" watchObservedRunningTime="2026-10-30 09:45:04.384321222 +0000 UTC m=+3068.107748365" Oct 30 09:45:05.170914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.170875 4920 generic.go:356] "Generic (PLEG): container finished" podID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerID="8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36" exitCode=60 Oct 30 09:45:05.171432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.170951 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerDied","Data":"8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36"} Oct 30 09:45:05.171432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.171060 4920 scope.go:122] "RemoveContainer" containerID="892c9f19239b132199e3ac4be374cf41996de0d57068124506ab967aa29f1cd1" Oct 30 09:45:05.171432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.171339 4920 scope.go:122] "RemoveContainer" containerID="8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36" Oct 30 09:45:05.171748 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:05.171723 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26_odh-ai-gateway-infra(5a865623-785c-4e1b-a6df-3748ff6d140f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" Oct 30 09:45:05.173367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.173341 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerStarted","Data":"20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529"} Oct 30 09:45:05.175289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.175270 4920 generic.go:356] "Generic (PLEG): container finished" podID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerID="b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358" exitCode=60 Oct 30 09:45:05.175383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.175344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerDied","Data":"b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358"} Oct 30 09:45:05.175949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.175616 4920 scope.go:122] "RemoveContainer" containerID="b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358" Oct 30 09:45:05.175949 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:05.175901 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s_odh-ai-gateway-infra(f2ad24d6-6372-486e-b990-5a448f1ad0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" Oct 30 09:45:05.177472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.177449 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerStarted","Data":"10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306"} Oct 30 09:45:05.179611 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.179593 4920 generic.go:356] "Generic (PLEG): container finished" podID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerID="d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83" exitCode=60 Oct 30 09:45:05.179716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.179662 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerDied","Data":"d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83"} Oct 30 09:45:05.179890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.179872 4920 scope.go:122] "RemoveContainer" containerID="d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83" Oct 30 09:45:05.180211 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:05.180114 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm_odh-ai-gateway-infra(2e0b0a9d-b7b8-411a-8f3e-7987a73438c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" Oct 30 09:45:05.181744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.181721 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerStarted","Data":"fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399"} Oct 30 09:45:05.183721 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.183691 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerStarted","Data":"fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e"} Oct 30 09:45:05.185498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.185475 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerStarted","Data":"cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d"} Oct 30 09:45:05.187343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.187321 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerStarted","Data":"5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf"} Oct 30 09:45:05.189099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.189083 4920 scope.go:122] "RemoveContainer" containerID="e5e7e8658c529a2a97ae4c345bcb54034069829997e6d88030d6b09d390c18f0" Oct 30 09:45:05.208370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.208206 4920 scope.go:122] "RemoveContainer" containerID="7935a6fa049c8974ad71879241a0cfb0dcae5897bb9be16b0bc9c5b5e38bb110" Oct 30 09:45:05.522583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:05.522535 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:06.198120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.198054 4920 scope.go:122] "RemoveContainer" containerID="8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36" Oct 30 09:45:06.198577 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.198517 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26_odh-ai-gateway-infra(5a865623-785c-4e1b-a6df-3748ff6d140f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" Oct 30 09:45:06.201950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.201915 4920 generic.go:356] "Generic (PLEG): container finished" podID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerID="20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529" exitCode=60 Oct 30 09:45:06.202104 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.202050 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerDied","Data":"20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529"} Oct 30 09:45:06.202149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.202107 4920 scope.go:122] "RemoveContainer" containerID="355f24f97a703c346ca041fe90de464b254f5262b0f367a281e33fe812e959f1" Oct 30 09:45:06.202600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.202582 4920 scope.go:122] "RemoveContainer" containerID="20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529" Oct 30 09:45:06.202972 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.202949 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9_odh-ai-gateway-infra(6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" Oct 30 09:45:06.205485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.205458 4920 scope.go:122] "RemoveContainer" containerID="b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358" Oct 30 09:45:06.205832 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.205807 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s_odh-ai-gateway-infra(f2ad24d6-6372-486e-b990-5a448f1ad0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" Oct 30 09:45:06.208086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.208058 4920 generic.go:356] "Generic (PLEG): container finished" podID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerID="10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306" exitCode=60 Oct 30 09:45:06.208193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.208161 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerDied","Data":"10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306"} Oct 30 09:45:06.208589 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.208568 4920 scope.go:122] "RemoveContainer" containerID="10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306" Oct 30 09:45:06.208909 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.208883 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m_odh-ai-gateway-infra(1efb2d2e-2f2b-4b2f-990e-83a91e0165f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" Oct 30 09:45:06.211577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.211558 4920 scope.go:122] "RemoveContainer" containerID="d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83" Oct 30 09:45:06.211864 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.211839 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm_odh-ai-gateway-infra(2e0b0a9d-b7b8-411a-8f3e-7987a73438c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" Oct 30 09:45:06.214447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.214420 4920 generic.go:356] "Generic (PLEG): container finished" podID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerID="fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399" exitCode=60 Oct 30 09:45:06.214558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.214534 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerDied","Data":"fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399"} Oct 30 09:45:06.214957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.214937 4920 scope.go:122] "RemoveContainer" containerID="fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399" Oct 30 09:45:06.215325 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.215300 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8_odh-ai-gateway-infra(a8aa4f39-3541-40e5-8209-4863af3c0a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" Oct 30 09:45:06.220169 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.220154 4920 scope.go:122] "RemoveContainer" containerID="320cf161ca9bec6149d8fd92d0b41b10a0376337dcb634fb2f5b6d788ac223ca" Oct 30 09:45:06.222550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.222523 4920 generic.go:356] "Generic (PLEG): container finished" podID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerID="fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e" exitCode=60 Oct 30 09:45:06.222668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.222601 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerDied","Data":"fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e"} Oct 30 09:45:06.222956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.222933 4920 scope.go:122] "RemoveContainer" containerID="fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e" Oct 30 09:45:06.223313 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.223289 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b_odh-ai-gateway-infra(0792860c-d44d-411d-b380-aeb7fcf9abe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" Oct 30 09:45:06.225923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.225109 4920 generic.go:356] "Generic (PLEG): container finished" podID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerID="cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d" exitCode=60 Oct 30 09:45:06.225923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.225253 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerDied","Data":"cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d"} Oct 30 09:45:06.225923 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.225496 4920 scope.go:122] "RemoveContainer" containerID="cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d" Oct 30 09:45:06.225923 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.225852 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r_odh-ai-gateway-infra(622f3ba5-37a8-4777-b470-ca325878fbcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" Oct 30 09:45:06.228124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.227366 4920 generic.go:356] "Generic (PLEG): container finished" podID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerID="5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf" exitCode=60 Oct 30 09:45:06.228124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.227416 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerDied","Data":"5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf"} Oct 30 09:45:06.228124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.227739 4920 scope.go:122] "RemoveContainer" containerID="5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf" Oct 30 09:45:06.228124 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:06.228074 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc_odh-ai-gateway-infra(49320e8c-dd17-474a-b5c0-8d2cec590723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" Oct 30 09:45:06.236797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.236675 4920 scope.go:122] "RemoveContainer" containerID="6ff6a8123c8fa83cf6d36f687a6dfc121216d4d32f1c1350f7fb995f906d2b07" Oct 30 09:45:06.255308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.255288 4920 scope.go:122] "RemoveContainer" containerID="a0c5fb28d7b78171329912b1c374da55d0591d2645dc6d8e39b000a71c1d08a8" Oct 30 09:45:06.269466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.269443 4920 scope.go:122] "RemoveContainer" containerID="c3f947f8ab339c78df7f7c70bbc6c7d0f886daee086b27d40424add7b26914da" Oct 30 09:45:06.286921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:06.286894 4920 scope.go:122] "RemoveContainer" containerID="5340bce152abccf9935016b3a47a6e684c92cc0a43cba8096be52c54273e58d1" Oct 30 09:45:07.017986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.017939 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:07.118702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.118669 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:07.239239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.236684 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:07.382789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.382757 4920 scope.go:122] "RemoveContainer" containerID="cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d" Oct 30 09:45:07.383207 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.383173 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r_odh-ai-gateway-infra(622f3ba5-37a8-4777-b470-ca325878fbcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" Oct 30 09:45:07.384919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.384894 4920 scope.go:122] "RemoveContainer" containerID="5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf" Oct 30 09:45:07.385509 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.385318 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc_odh-ai-gateway-infra(49320e8c-dd17-474a-b5c0-8d2cec590723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" Oct 30 09:45:07.386549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.386505 4920 scope.go:122] "RemoveContainer" containerID="20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529" Oct 30 09:45:07.386818 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.386795 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9_odh-ai-gateway-infra(6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" Oct 30 09:45:07.388227 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.388196 4920 scope.go:122] "RemoveContainer" containerID="10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306" Oct 30 09:45:07.388549 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.388527 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m_odh-ai-gateway-infra(1efb2d2e-2f2b-4b2f-990e-83a91e0165f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" Oct 30 09:45:07.389768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.389752 4920 scope.go:122] "RemoveContainer" containerID="fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399" Oct 30 09:45:07.389962 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.389945 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8_odh-ai-gateway-infra(a8aa4f39-3541-40e5-8209-4863af3c0a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" Oct 30 09:45:07.391976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:07.391960 4920 scope.go:122] "RemoveContainer" containerID="fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e" Oct 30 09:45:07.392165 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:07.392149 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b_odh-ai-gateway-infra(0792860c-d44d-411d-b380-aeb7fcf9abe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" Oct 30 09:45:08.652443 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:08.652407 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:08.957378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:08.957283 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:08.957585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:08.957367 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:09.413725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.413683 4920 generic.go:356] "Generic (PLEG): container finished" podID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerID="6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083" exitCode=0 Oct 30 09:45:09.413928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.413758 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerDied","Data":"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083"} Oct 30 09:45:09.414420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.414399 4920 scope.go:122] "RemoveContainer" containerID="6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083" Oct 30 09:45:09.416432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.416405 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:45:09.416509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.416471 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:45:09.416561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:09.416507 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:45:10.247085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:10.247042 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:10.456533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:10.456493 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerStarted","Data":"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7"} Oct 30 09:45:11.415240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:11.408621 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:45:11.415240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:11.408751 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:11.415240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:11.408793 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:11.768858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:11.768773 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:12.397183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:12.397147 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:12.397422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:12.397209 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:12.953432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:12.953399 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.058430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.058390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.162483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.162445 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.276049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.275967 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.421241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.420400 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:13.421241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.420454 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:13.421813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.421690 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.653733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.653702 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.760562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.760522 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:13.867371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:13.867332 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:14.402244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:14.401868 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:14.402244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:14.401922 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:14.496616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:14.496579 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:14.594086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:14.594052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:15.343895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:15.343857 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:15.410712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:15.410675 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:15.411319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:15.411284 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:15.761310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:15.761235 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:15.861459 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:15.861421 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:16.045031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.044983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.426008 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.426078 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.426480 4920 scope.go:122] "RemoveContainer" containerID="5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf" Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.426782 4920 scope.go:122] "RemoveContainer" containerID="cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d" Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.427699 4920 scope.go:122] "RemoveContainer" containerID="20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529" Oct 30 09:45:16.429652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:16.429298 4920 scope.go:122] "RemoveContainer" containerID="10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306" Oct 30 09:45:17.403887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.403854 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:45:17.404332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.404245 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:45:17.405052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.405035 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:45:17.443328 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.443294 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:17.733487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.733450 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerStarted","Data":"9003d7e2d9d2ffbe54abeb5ab71d310683816460bc479c4c7106fb0df0e5d83f"} Oct 30 09:45:17.735511 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.735488 4920 generic.go:356] "Generic (PLEG): container finished" podID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerID="c92bd5f21bbcef901ff9e23875ced0624ce5779ecc24da332db8b66453eb41a2" exitCode=60 Oct 30 09:45:17.735617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.735557 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerDied","Data":"c92bd5f21bbcef901ff9e23875ced0624ce5779ecc24da332db8b66453eb41a2"} Oct 30 09:45:17.735673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.735605 4920 scope.go:122] "RemoveContainer" containerID="20bd96e593ac7ddb7083f7a103df5ef0b1cd6c2b98b00f6ac2dfd5ed12803529" Oct 30 09:45:17.735974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.735956 4920 scope.go:122] "RemoveContainer" containerID="c92bd5f21bbcef901ff9e23875ced0624ce5779ecc24da332db8b66453eb41a2" Oct 30 09:45:17.736317 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:17.736292 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9_odh-ai-gateway-infra(6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" Oct 30 09:45:17.737808 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.737787 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerStarted","Data":"0df395446c9dee1ddfa8671721c532027c647647a8184cb9f07e6585825080bd"} Oct 30 09:45:17.740981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:17.740954 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerStarted","Data":"def2d4ec6312b80ac1df33b96f07536788579b99da0f655f66a275ecf6bc6e57"} Oct 30 09:45:18.391185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.391146 4920 scope.go:122] "RemoveContainer" containerID="d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83" Oct 30 09:45:18.391610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.391587 4920 scope.go:122] "RemoveContainer" containerID="b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358" Oct 30 09:45:18.391732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.391674 4920 scope.go:122] "RemoveContainer" containerID="8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36" Oct 30 09:45:18.767122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.767031 4920 generic.go:356] "Generic (PLEG): container finished" podID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerID="0df395446c9dee1ddfa8671721c532027c647647a8184cb9f07e6585825080bd" exitCode=60 Oct 30 09:45:18.767122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.767098 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerDied","Data":"0df395446c9dee1ddfa8671721c532027c647647a8184cb9f07e6585825080bd"} Oct 30 09:45:18.767663 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.767191 4920 scope.go:122] "RemoveContainer" containerID="10ec8f3651c009421b3b5ac4418edbd2ff4c106bded77c3eb55cd44593ee1306" Oct 30 09:45:18.767732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.767658 4920 scope.go:122] "RemoveContainer" containerID="0df395446c9dee1ddfa8671721c532027c647647a8184cb9f07e6585825080bd" Oct 30 09:45:18.768054 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:18.768029 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m_odh-ai-gateway-infra(1efb2d2e-2f2b-4b2f-990e-83a91e0165f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" Oct 30 09:45:18.769479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.769460 4920 generic.go:356] "Generic (PLEG): container finished" podID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerID="def2d4ec6312b80ac1df33b96f07536788579b99da0f655f66a275ecf6bc6e57" exitCode=60 Oct 30 09:45:18.769573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.769527 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerDied","Data":"def2d4ec6312b80ac1df33b96f07536788579b99da0f655f66a275ecf6bc6e57"} Oct 30 09:45:18.769889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.769870 4920 scope.go:122] "RemoveContainer" containerID="def2d4ec6312b80ac1df33b96f07536788579b99da0f655f66a275ecf6bc6e57" Oct 30 09:45:18.770225 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:18.770189 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r_odh-ai-gateway-infra(622f3ba5-37a8-4777-b470-ca325878fbcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" Oct 30 09:45:18.771358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.771339 4920 generic.go:356] "Generic (PLEG): container finished" podID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerID="9003d7e2d9d2ffbe54abeb5ab71d310683816460bc479c4c7106fb0df0e5d83f" exitCode=60 Oct 30 09:45:18.771429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.771406 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerDied","Data":"9003d7e2d9d2ffbe54abeb5ab71d310683816460bc479c4c7106fb0df0e5d83f"} Oct 30 09:45:18.771843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.771829 4920 scope.go:122] "RemoveContainer" containerID="9003d7e2d9d2ffbe54abeb5ab71d310683816460bc479c4c7106fb0df0e5d83f" Oct 30 09:45:18.772076 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:18.772061 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc_odh-ai-gateway-infra(49320e8c-dd17-474a-b5c0-8d2cec590723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" Oct 30 09:45:18.782909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.782882 4920 scope.go:122] "RemoveContainer" containerID="cdf542cfc745d0d1a2354b2026a75be4b45632b3566fbdeefd4abb05009c0b9d" Oct 30 09:45:18.787024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.787001 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:18.801412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.801385 4920 scope.go:122] "RemoveContainer" containerID="5a66bc36418b563f8535e2e20851eff5868f29725f583778c8387367b65ab0cf" Oct 30 09:45:18.820472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.820447 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:18.870474 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.870441 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:18.913369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:18.912830 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:19.107913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.107883 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:19.109227 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.109187 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:19.247694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.247667 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\") pod \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " Oct 30 09:45:19.248274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.248253 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\" (UniqueName: \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\") pod \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\" (UID: \"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d\") " Oct 30 09:45:19.248690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.248194 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca" pod "6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" (UID: "6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.249418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.249399 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.254775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.254738 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6" pod "6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" (UID: "6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d"). InnerVolumeSpecName "kube-api-access-pt7l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.351166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.351052 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7l6\" (UniqueName: \"kubernetes.io/projected/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d-kube-api-access-pt7l6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.391546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.391511 4920 scope.go:122] "RemoveContainer" containerID="fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e" Oct 30 09:45:19.391735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.391693 4920 scope.go:122] "RemoveContainer" containerID="fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399" Oct 30 09:45:19.786532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.786493 4920 generic.go:356] "Generic (PLEG): container finished" podID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerID="0ce8fbaf3e9babf8a1e4bec09898c785e1525ebdbda9f015ba2cd4aa37b6d7ca" exitCode=60 Oct 30 09:45:19.786969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.786553 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerDied","Data":"0ce8fbaf3e9babf8a1e4bec09898c785e1525ebdbda9f015ba2cd4aa37b6d7ca"} Oct 30 09:45:19.786969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.786612 4920 scope.go:122] "RemoveContainer" containerID="8a9b0fc0698c0029efb4e9a0cb801c9c048ac3ccd95fd706a4151eaa7a110a36" Oct 30 09:45:19.787120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.787099 4920 scope.go:122] "RemoveContainer" containerID="0ce8fbaf3e9babf8a1e4bec09898c785e1525ebdbda9f015ba2cd4aa37b6d7ca" Oct 30 09:45:19.787592 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:19.787558 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26_odh-ai-gateway-infra(5a865623-785c-4e1b-a6df-3748ff6d140f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" Oct 30 09:45:19.788477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.788449 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" event={"ID":"6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d","Type":"ContainerDied","Data":"30a6ac3b8dd4fb1bba45ac27fd343e9be1f3b9899ba5182499e4a4846c66baba"} Oct 30 09:45:19.788607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.788535 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9" Oct 30 09:45:19.793516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.793488 4920 generic.go:356] "Generic (PLEG): container finished" podID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerID="750f9c24c6febbac817db766b52a635bcb230a60a157417a8fa7076f22a53c6e" exitCode=60 Oct 30 09:45:19.793624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.793537 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerDied","Data":"750f9c24c6febbac817db766b52a635bcb230a60a157417a8fa7076f22a53c6e"} Oct 30 09:45:19.794040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.794021 4920 scope.go:122] "RemoveContainer" containerID="750f9c24c6febbac817db766b52a635bcb230a60a157417a8fa7076f22a53c6e" Oct 30 09:45:19.794463 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:19.794435 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s_odh-ai-gateway-infra(f2ad24d6-6372-486e-b990-5a448f1ad0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" Oct 30 09:45:19.801550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.799252 4920 generic.go:356] "Generic (PLEG): container finished" podID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerID="6fa94889381d30983f1edb52b13792cbbc5b1b71dc03c564aca2933e3bd74c5c" exitCode=60 Oct 30 09:45:19.801550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.799322 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerDied","Data":"6fa94889381d30983f1edb52b13792cbbc5b1b71dc03c564aca2933e3bd74c5c"} Oct 30 09:45:19.801550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.800776 4920 scope.go:122] "RemoveContainer" containerID="6fa94889381d30983f1edb52b13792cbbc5b1b71dc03c564aca2933e3bd74c5c" Oct 30 09:45:19.801550 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:19.801528 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm_odh-ai-gateway-infra(2e0b0a9d-b7b8-411a-8f3e-7987a73438c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" Oct 30 09:45:19.812168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.812144 4920 scope.go:122] "RemoveContainer" containerID="c92bd5f21bbcef901ff9e23875ced0624ce5779ecc24da332db8b66453eb41a2" Oct 30 09:45:19.830262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.830232 4920 scope.go:122] "RemoveContainer" containerID="b2849c7946c833f6f69962c74f1dd8d10c95b4a1ab158e551a12b121d4717358" Oct 30 09:45:19.850884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:19.850842 4920 scope.go:122] "RemoveContainer" containerID="d79c82265f24c87236a34e5fa72078de1e88dbb9e6cb54e9a6d49cae03317e83" Oct 30 09:45:20.064454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.064425 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:20.139813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.139752 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:20.143457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.143431 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:20.170496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170464 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\") pod \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " Oct 30 09:45:20.170670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170514 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\") pod \"622f3ba5-37a8-4777-b470-ca325878fbcf\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " Oct 30 09:45:20.170670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170565 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\" (UniqueName: \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\") pod \"49320e8c-dd17-474a-b5c0-8d2cec590723\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " Oct 30 09:45:20.170670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170619 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\" (UniqueName: \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\") pod \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\" (UID: \"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3\") " Oct 30 09:45:20.170843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170710 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\") pod \"622f3ba5-37a8-4777-b470-ca325878fbcf\" (UID: \"622f3ba5-37a8-4777-b470-ca325878fbcf\") " Oct 30 09:45:20.170908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170882 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\") pod \"49320e8c-dd17-474a-b5c0-8d2cec590723\" (UID: \"49320e8c-dd17-474a-b5c0-8d2cec590723\") " Oct 30 09:45:20.170969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170915 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca" pod "1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" (UID: "1efb2d2e-2f2b-4b2f-990e-83a91e0165f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.170969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.170924 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca" pod "622f3ba5-37a8-4777-b470-ca325878fbcf" (UID: "622f3ba5-37a8-4777-b470-ca325878fbcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.171399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.171373 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca" pod "49320e8c-dd17-474a-b5c0-8d2cec590723" (UID: "49320e8c-dd17-474a-b5c0-8d2cec590723"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.171631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.171612 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49320e8c-dd17-474a-b5c0-8d2cec590723-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.171700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.171636 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.171700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.171650 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f3ba5-37a8-4777-b470-ca325878fbcf-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.172941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.172923 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx" pod "49320e8c-dd17-474a-b5c0-8d2cec590723" (UID: "49320e8c-dd17-474a-b5c0-8d2cec590723"). InnerVolumeSpecName "kube-api-access-n5lvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.172993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.172942 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt" pod "622f3ba5-37a8-4777-b470-ca325878fbcf" (UID: "622f3ba5-37a8-4777-b470-ca325878fbcf"). InnerVolumeSpecName "kube-api-access-bwjlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.173140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.173126 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485" pod "1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" (UID: "1efb2d2e-2f2b-4b2f-990e-83a91e0165f3"). InnerVolumeSpecName "kube-api-access-94485". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.206398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.206359 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:20.210786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.210606 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d66d22-29889225-rsbv9"] Oct 30 09:45:20.272802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.272763 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lvx\" (UniqueName: \"kubernetes.io/projected/49320e8c-dd17-474a-b5c0-8d2cec590723-kube-api-access-n5lvx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.272802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.272801 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94485\" (UniqueName: \"kubernetes.io/projected/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3-kube-api-access-94485\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.273055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.272816 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjlt\" (UniqueName: \"kubernetes.io/projected/622f3ba5-37a8-4777-b470-ca325878fbcf-kube-api-access-bwjlt\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.401446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.401353 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" path="/var/lib/kubelet/pods/6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d/volumes" Oct 30 09:45:20.812636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.812606 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" Oct 30 09:45:20.812636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.812617 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m" event={"ID":"1efb2d2e-2f2b-4b2f-990e-83a91e0165f3","Type":"ContainerDied","Data":"15cee9e57336382538b649cb433e3c7922d2bc0fdeb439b0fac90509907b153b"} Oct 30 09:45:20.813188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.812701 4920 scope.go:122] "RemoveContainer" containerID="0df395446c9dee1ddfa8671721c532027c647647a8184cb9f07e6585825080bd" Oct 30 09:45:20.816830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.816806 4920 generic.go:356] "Generic (PLEG): container finished" podID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerID="a7884345c5dc2096f888a7664baf30cb585bb247c6bfc38f271b29a09ce754fe" exitCode=60 Oct 30 09:45:20.816951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.816879 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerDied","Data":"a7884345c5dc2096f888a7664baf30cb585bb247c6bfc38f271b29a09ce754fe"} Oct 30 09:45:20.817386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.817369 4920 scope.go:122] "RemoveContainer" containerID="a7884345c5dc2096f888a7664baf30cb585bb247c6bfc38f271b29a09ce754fe" Oct 30 09:45:20.817695 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:20.817677 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8_odh-ai-gateway-infra(a8aa4f39-3541-40e5-8209-4863af3c0a73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" Oct 30 09:45:20.818918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.818897 4920 generic.go:356] "Generic (PLEG): container finished" podID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerID="3f5ae5a082992944fe9e1c0a52146b35547ce05da5ed8acc615781e9441e0bd9" exitCode=60 Oct 30 09:45:20.819032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.818960 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerDied","Data":"3f5ae5a082992944fe9e1c0a52146b35547ce05da5ed8acc615781e9441e0bd9"} Oct 30 09:45:20.819361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.819343 4920 scope.go:122] "RemoveContainer" containerID="3f5ae5a082992944fe9e1c0a52146b35547ce05da5ed8acc615781e9441e0bd9" Oct 30 09:45:20.819652 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:20.819629 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b_odh-ai-gateway-infra(0792860c-d44d-411d-b380-aeb7fcf9abe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" Oct 30 09:45:20.820448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.820430 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" Oct 30 09:45:20.820554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.820457 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r" event={"ID":"622f3ba5-37a8-4777-b470-ca325878fbcf","Type":"ContainerDied","Data":"a0fa9d5cd227a10f99e3a6c20bf619de40ddab4475860e61818218f149c23b42"} Oct 30 09:45:20.822034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.822008 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" Oct 30 09:45:20.822170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.822042 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc" event={"ID":"49320e8c-dd17-474a-b5c0-8d2cec590723","Type":"ContainerDied","Data":"9669985a0c104531286b27daa4931ce376e4c54855621dc2d8011d52df3b751d"} Oct 30 09:45:20.826013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.825993 4920 scope.go:122] "RemoveContainer" containerID="fbc50feaed72a8625e99546b8c42a846c8f0ce863ca1f03d00b2fe7fdc4b9399" Oct 30 09:45:20.840656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.840633 4920 scope.go:122] "RemoveContainer" containerID="fb8746bf40c2b698ffe0e960b1ea6e84be92971a472c84b6f116d52d261c9e2e" Oct 30 09:45:20.855333 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.855308 4920 scope.go:122] "RemoveContainer" containerID="def2d4ec6312b80ac1df33b96f07536788579b99da0f655f66a275ecf6bc6e57" Oct 30 09:45:20.865514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.865489 4920 scope.go:122] "RemoveContainer" containerID="9003d7e2d9d2ffbe54abeb5ab71d310683816460bc479c4c7106fb0df0e5d83f" Oct 30 09:45:20.876587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.876556 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:20.916910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.916841 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:20.941040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.941005 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889225-zph7m"] Oct 30 09:45:20.956957 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.956921 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:20.968796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.968769 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:20.979922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.977352 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889225-vzvqc"] Oct 30 09:45:20.998668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:20.998341 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:21.011272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.011192 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:21.018794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.018725 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae795b-29889225-cdg7r"] Oct 30 09:45:21.111802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.108419 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:21.177551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.177469 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:21.298559 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.298511 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\" (UniqueName: \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\") pod \"5a865623-785c-4e1b-a6df-3748ff6d140f\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " Oct 30 09:45:21.298777 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.298595 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\") pod \"5a865623-785c-4e1b-a6df-3748ff6d140f\" (UID: \"5a865623-785c-4e1b-a6df-3748ff6d140f\") " Oct 30 09:45:21.299445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.299418 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca" pod "5a865623-785c-4e1b-a6df-3748ff6d140f" (UID: "5a865623-785c-4e1b-a6df-3748ff6d140f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.302759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.302736 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm" pod "5a865623-785c-4e1b-a6df-3748ff6d140f" (UID: "5a865623-785c-4e1b-a6df-3748ff6d140f"). InnerVolumeSpecName "kube-api-access-hjnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.317552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.317370 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:21.339485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.339455 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:21.400788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.400750 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjnjm\" (UniqueName: \"kubernetes.io/projected/5a865623-785c-4e1b-a6df-3748ff6d140f-kube-api-access-hjnjm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.400788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.400788 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a865623-785c-4e1b-a6df-3748ff6d140f-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.501644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.501565 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\") pod \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " Oct 30 09:45:21.501644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.501622 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\") pod \"f2ad24d6-6372-486e-b990-5a448f1ad0af\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " Oct 30 09:45:21.501644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.501645 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\" (UniqueName: \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\") pod \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\" (UID: \"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9\") " Oct 30 09:45:21.501955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.501693 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\" (UniqueName: \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\") pod \"f2ad24d6-6372-486e-b990-5a448f1ad0af\" (UID: \"f2ad24d6-6372-486e-b990-5a448f1ad0af\") " Oct 30 09:45:21.502073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.502047 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca" pod "2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" (UID: "2e0b0a9d-b7b8-411a-8f3e-7987a73438c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.502151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.502057 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca" pod "f2ad24d6-6372-486e-b990-5a448f1ad0af" (UID: "f2ad24d6-6372-486e-b990-5a448f1ad0af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.502672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.502646 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.502672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.502675 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ad24d6-6372-486e-b990-5a448f1ad0af-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.504145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.504113 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9" pod "f2ad24d6-6372-486e-b990-5a448f1ad0af" (UID: "f2ad24d6-6372-486e-b990-5a448f1ad0af"). InnerVolumeSpecName "kube-api-access-jm5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.504294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.504243 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl" pod "2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" (UID: "2e0b0a9d-b7b8-411a-8f3e-7987a73438c9"). InnerVolumeSpecName "kube-api-access-pv6wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.603991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.603953 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6wl\" (UniqueName: \"kubernetes.io/projected/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9-kube-api-access-pv6wl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.603991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.603986 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm5r9\" (UniqueName: \"kubernetes.io/projected/f2ad24d6-6372-486e-b990-5a448f1ad0af-kube-api-access-jm5r9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.835840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.835805 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" Oct 30 09:45:21.835840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.835820 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm" event={"ID":"2e0b0a9d-b7b8-411a-8f3e-7987a73438c9","Type":"ContainerDied","Data":"8f55c3975da830560a1b1600cce3a71972188c9c39e8e164baf80af6f21fda76"} Oct 30 09:45:21.836414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.835888 4920 scope.go:122] "RemoveContainer" containerID="6fa94889381d30983f1edb52b13792cbbc5b1b71dc03c564aca2933e3bd74c5c" Oct 30 09:45:21.851562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.846647 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" Oct 30 09:45:21.851562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.846644 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26" event={"ID":"5a865623-785c-4e1b-a6df-3748ff6d140f","Type":"ContainerDied","Data":"c25da19e071b74625da051120fa045e263e91d228a2fd8c05419cdc18eee8fea"} Oct 30 09:45:21.851562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.848365 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" event={"ID":"f2ad24d6-6372-486e-b990-5a448f1ad0af","Type":"ContainerDied","Data":"67d5ee120d15b8e66fba1130860d8d45ffecaca8d8537816c3e06447c4746d89"} Oct 30 09:45:21.851562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.848454 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s" Oct 30 09:45:21.853479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.853450 4920 scope.go:122] "RemoveContainer" containerID="0ce8fbaf3e9babf8a1e4bec09898c785e1525ebdbda9f015ba2cd4aa37b6d7ca" Oct 30 09:45:21.867883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.867861 4920 scope.go:122] "RemoveContainer" containerID="750f9c24c6febbac817db766b52a635bcb230a60a157417a8fa7076f22a53c6e" Oct 30 09:45:21.884785 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.884755 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:21.909009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.908966 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:21.913233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.913167 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c88ae-29889225-snzjm"] Oct 30 09:45:21.966736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.964750 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:21.970355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:21.970281 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-19c94a-29889225-mtm26"] Oct 30 09:45:22.014974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.014936 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:22.021442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.021402 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-304459-29889225-d674s"] Oct 30 09:45:22.064167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.064134 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:22.071471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.071445 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:22.226280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.215115 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\" (UniqueName: \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\") pod \"0792860c-d44d-411d-b380-aeb7fcf9abe7\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " Oct 30 09:45:22.226280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.215206 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\") pod \"0792860c-d44d-411d-b380-aeb7fcf9abe7\" (UID: \"0792860c-d44d-411d-b380-aeb7fcf9abe7\") " Oct 30 09:45:22.226280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.216625 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca" pod "0792860c-d44d-411d-b380-aeb7fcf9abe7" (UID: "0792860c-d44d-411d-b380-aeb7fcf9abe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:22.229931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.229764 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77" pod "0792860c-d44d-411d-b380-aeb7fcf9abe7" (UID: "0792860c-d44d-411d-b380-aeb7fcf9abe7"). InnerVolumeSpecName "kube-api-access-c5p77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:22.255019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.254987 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:22.316960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.316926 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\" (UniqueName: \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\") pod \"a8aa4f39-3541-40e5-8209-4863af3c0a73\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " Oct 30 09:45:22.317170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.317028 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\") pod \"a8aa4f39-3541-40e5-8209-4863af3c0a73\" (UID: \"a8aa4f39-3541-40e5-8209-4863af3c0a73\") " Oct 30 09:45:22.317670 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.317648 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5p77\" (UniqueName: \"kubernetes.io/projected/0792860c-d44d-411d-b380-aeb7fcf9abe7-kube-api-access-c5p77\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:22.317749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.317671 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0792860c-d44d-411d-b380-aeb7fcf9abe7-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:22.318276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.318251 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca" pod "a8aa4f39-3541-40e5-8209-4863af3c0a73" (UID: "a8aa4f39-3541-40e5-8209-4863af3c0a73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:22.320039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.320011 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh" pod "a8aa4f39-3541-40e5-8209-4863af3c0a73" (UID: "a8aa4f39-3541-40e5-8209-4863af3c0a73"). InnerVolumeSpecName "kube-api-access-vsllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:22.400451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.400371 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" path="/var/lib/kubelet/pods/1efb2d2e-2f2b-4b2f-990e-83a91e0165f3/volumes" Oct 30 09:45:22.401018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.401002 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" path="/var/lib/kubelet/pods/2e0b0a9d-b7b8-411a-8f3e-7987a73438c9/volumes" Oct 30 09:45:22.401577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.401561 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" path="/var/lib/kubelet/pods/49320e8c-dd17-474a-b5c0-8d2cec590723/volumes" Oct 30 09:45:22.402544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.402526 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" path="/var/lib/kubelet/pods/5a865623-785c-4e1b-a6df-3748ff6d140f/volumes" Oct 30 09:45:22.403063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.403047 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" path="/var/lib/kubelet/pods/622f3ba5-37a8-4777-b470-ca325878fbcf/volumes" Oct 30 09:45:22.403601 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.403585 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" path="/var/lib/kubelet/pods/f2ad24d6-6372-486e-b990-5a448f1ad0af/volumes" Oct 30 09:45:22.419377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.419347 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsllh\" (UniqueName: \"kubernetes.io/projected/a8aa4f39-3541-40e5-8209-4863af3c0a73-kube-api-access-vsllh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:22.419377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.419373 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa4f39-3541-40e5-8209-4863af3c0a73-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:45:22.609984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.609943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:22.867132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.867080 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" Oct 30 09:45:22.867132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.867092 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b" event={"ID":"0792860c-d44d-411d-b380-aeb7fcf9abe7","Type":"ContainerDied","Data":"5d411b021fce750895752201a0ce886935cf4535f83795327e6e018ed2219c01"} Oct 30 09:45:22.867772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.867165 4920 scope.go:122] "RemoveContainer" containerID="3f5ae5a082992944fe9e1c0a52146b35547ce05da5ed8acc615781e9441e0bd9" Oct 30 09:45:22.873625 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.873588 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" event={"ID":"a8aa4f39-3541-40e5-8209-4863af3c0a73","Type":"ContainerDied","Data":"5f68ee4d8b3e8e0f80fed36aa2f40bb82bc6aa0ce974695e59bdf4d0cf5a295d"} Oct 30 09:45:22.873756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.873607 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8" Oct 30 09:45:22.879597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.879573 4920 scope.go:122] "RemoveContainer" containerID="a7884345c5dc2096f888a7664baf30cb585bb247c6bfc38f271b29a09ce754fe" Oct 30 09:45:22.926468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.926426 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:22.932839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.932805 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-881f83-29889225-5gfh8"] Oct 30 09:45:22.959988 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.959955 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:22.964444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:22.964418 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7a213d-29889225-qwf5b"] Oct 30 09:45:24.269753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:24.269715 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:24.428967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:24.428921 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" path="/var/lib/kubelet/pods/0792860c-d44d-411d-b380-aeb7fcf9abe7/volumes" Oct 30 09:45:24.429566 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:24.429521 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" path="/var/lib/kubelet/pods/a8aa4f39-3541-40e5-8209-4863af3c0a73/volumes" Oct 30 09:45:25.887887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:25.887856 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:26.701503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:26.701470 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:27.371174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:27.371136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:27.935717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:27.935676 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:28.036645 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:28.036609 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:28.259177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:28.259094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:28.546074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:28.546037 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:28.701314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:28.701280 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:28.804592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:28.804508 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:29.088856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:29.088735 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:29.371429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:29.371357 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:29.469629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:29.469565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:30.672158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:30.672089 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:31.140134 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:45:31.140093 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:45:32.157699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:32.157662 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:33.340013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:33.339974 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:33.632627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:33.632544 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:35.273793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:35.273756 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:36.270995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:36.270955 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:45:36.313755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:36.313712 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:36.440382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:36.440342 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:36.998805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:36.998767 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:37.110548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:37.110502 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:38.889627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:38.889537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:40.646116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:40.646081 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:42.258877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:42.258839 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:43.073978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:43.073941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:43.185477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:43.185424 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:43.280387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:43.280345 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:43.760544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:43.760507 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:45.285388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:45.285346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:46.941587 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:46.941545 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:48.424057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:48.424018 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:48.956976 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:48.956941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:49.060285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:49.060247 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:49.169571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:49.169536 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:49.481492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:49.481439 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:49.584004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:49.583950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:49.867101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:49.867059 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:50.262493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:50.262401 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:50.546243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:50.541500 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:50.640874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:50.640833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:51.973070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:51.973018 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:52.072898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:52.072841 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:52.684866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:52.684809 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:53.074965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:53.074918 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:53.275241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:53.269664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:53.513128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:53.513048 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:53.643302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:53.633067 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:54.010936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:54.010898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:55.564499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:55.564449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:57.170953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:57.170901 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:45:58.879972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:45:58.879934 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:00.357026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:00.356965 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:01.138434 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:46:01.138393 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:46:01.845388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:01.845329 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:03.400966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:03.400904 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:04.914674 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:04.914643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:06.463868 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:06.463831 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:07.992076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:07.992037 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:09.678250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:09.678175 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:10.775930 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.775894 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:46:10.776616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776598 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776617 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776628 4920 cpu_manager.go:414] "containerMap: removing container" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776635 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776644 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776649 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776655 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776659 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.776664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776667 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776671 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776681 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776686 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776694 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776698 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776709 4920 cpu_manager.go:414] "containerMap: removing container" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776715 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776722 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776726 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776731 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776736 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776742 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776745 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776754 4920 cpu_manager.go:414] "containerMap: removing container" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776758 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776768 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776772 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776779 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776783 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776791 4920 cpu_manager.go:414] "containerMap: removing container" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776795 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776801 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776806 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776815 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776819 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776827 4920 cpu_manager.go:414] "containerMap: removing container" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776831 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776846 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776851 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776859 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776863 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776869 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.776927 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776873 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776966 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776976 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776982 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776988 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776993 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.776998 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777003 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777008 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777014 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777018 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777024 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777029 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777035 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777039 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777045 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777060 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777065 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777071 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777079 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777085 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:46:10.777978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.777090 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:46:10.782392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.782368 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.842035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.841991 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842357 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842471 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842527 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842610 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842640 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842692 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvptc\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842867 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.846179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.842929 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.848390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.848158 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:46:10.944460 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944421 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.944680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944480 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.944680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944563 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.944680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944580 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.944680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944618 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvptc\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944716 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944749 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944810 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.944872 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.945183 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.945254 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.945298 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.945384 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.945602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.945572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.947326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.947304 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.947560 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.947542 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.964680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.964652 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:10.964793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:10.964754 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvptc\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\") pod \"maas-default-gateway-openshift-default-77f556d647-lpk2w\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:11.102008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.101965 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:11.369681 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.369603 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:11.485804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.484864 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.485804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.484964 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.485804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.485007 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.488802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.488762 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:46:11.545389 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:11.545353 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" event={"ID":"00c7162e-8cce-4604-a97e-057171973a45","Type":"ContainerStarted","Data":"322b91e7606533b34f04cd84af029bcabd0cb21e847a3dd9190af54046c058e1"} Oct 30 09:46:12.191850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:12.191812 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:46:12.557408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:12.557367 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" event={"ID":"00c7162e-8cce-4604-a97e-057171973a45","Type":"ContainerStarted","Data":"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6"} Oct 30 09:46:13.069193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:13.069159 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:13.108763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:13.106598 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:13.119241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:13.112345 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:13.119241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:13.112406 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:14.119242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:14.118693 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:14.119242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:14.118769 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:14.777067 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:14.777033 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:15.106243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:15.102724 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:15.106243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:15.102792 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:16.111241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:16.103608 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:16.111241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:16.103668 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:16.370251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:16.370150 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:16.673404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:16.673315 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:17.102688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:17.102647 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:17.102987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:17.102957 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:18.102975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:18.102930 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:18.103530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:18.102982 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:18.353810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:18.353716 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:19.102725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:19.102676 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lpk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 09:46:19.102938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:19.102752 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 09:46:19.859009 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:19.858968 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:20.464997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:20.464949 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:20.465431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:20.465396 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:20.467340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:20.467313 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:46:20.526005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:20.525938 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podStartSLOduration=10.525918052 podStartE2EDuration="10.525918052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:12.622796546 +0000 UTC m=+3136.346223690" watchObservedRunningTime="2026-10-30 09:46:20.525918052 +0000 UTC m=+3144.249345195" Oct 30 09:46:21.479963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:21.479928 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:23.057567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:23.057525 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:24.663465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:24.663421 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:26.282242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:26.282193 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:27.770886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:27.770849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:29.253254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:29.253196 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:30.811100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:30.811061 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:31.138822 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:46:31.138713 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:46:32.366899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:32.366854 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:33.866956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:33.866924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:35.352126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:35.352079 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:36.915693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:36.915652 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:38.331517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:38.331472 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:38.466252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:38.457135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:38.784664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:38.784634 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:39.446199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:39.446160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:39.551687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:39.551648 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:40.789365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:40.789285 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:42.386803 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:42.386770 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:43.966544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:43.966496 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:45.460096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:45.460053 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:47.034577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:47.034533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:48.520011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:48.519972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:50.062415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:50.062372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:51.741486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:51.741454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:52.520877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:52.520838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:52.590961 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:52.590927 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:46:52.622654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:52.622619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:52.955652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:52.955613 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:53.052847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:53.052814 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:53.153153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:53.153117 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:53.807915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:53.807881 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:55.266873 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:55.266838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:56.778004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:56.777971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:58.345393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:58.345359 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:59.255827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:59.255794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:59.364183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:59.358820 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:59.455598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:59.455560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:46:59.959963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:46:59.959931 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:01.139413 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:47:01.139366 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:47:01.536279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:01.536238 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:03.073978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:03.073943 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:04.638858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:04.638825 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:05.249359 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:05.249325 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:05.350841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:05.350806 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:06.224202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:06.224167 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:06.328484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:06.328452 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:08.069661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:08.069620 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:09.654558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:09.654523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:11.193899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:11.193861 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:14.191819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:14.191785 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:14.356813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:14.356773 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:15.970593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:15.970555 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:17.464099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:17.464050 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:19.048136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:19.044802 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:20.673910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:20.673867 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:22.170565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:22.170530 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:23.767786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:23.767687 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:25.593119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:25.593074 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:27.499542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:27.496327 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:27.504964 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:27.503608 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:27.505196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:27.505162 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:27.621063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:27.620992 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:27.634280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:27.632316 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:28.566388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:28.566349 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:28.567431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:28.567408 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:47:28.846303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:28.844714 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" Oct 30 09:47:29.347614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:29.347573 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:30.961947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:30.961865 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:31.136859 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:47:31.136824 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:47:32.601289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:32.601254 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:34.230778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:34.230745 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:35.704368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:35.704333 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:37.265432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:37.265360 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:38.848820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:38.848765 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:40.463018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:40.462985 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:42.021551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:42.021512 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:43.018862 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:43.018828 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:47:43.571021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:43.570956 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:45.419178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:45.419137 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:47.497240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:47.493990 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:47:47.497240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:47.496610 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:48.994773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:48.994737 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:50.570905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:50.570867 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:50.827232 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:50.827129 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:50.942239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:50.940733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:51.955764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:51.955728 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:52.054949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:52.054918 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:52.261466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:52.261377 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:52.576272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:52.576236 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:53.102760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:53.102718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:53.202367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:53.202328 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:53.918890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:53.918830 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:55.477307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:55.477273 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:57.044056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:57.044021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:58.600541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:58.600491 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:58.732089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:58.729480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:58.823301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:58.823259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:47:58.922832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:47:58.922748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:00.280713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:00.280681 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:01.135953 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:48:01.135908 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:48:01.786056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:01.786018 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:03.267909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:03.267871 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:03.396616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:03.396553 4920 scope.go:122] "RemoveContainer" containerID="36b7f58631c8c5f98591b1beaee5e3fc6927eb2d4657a10a731ff43bf332fe90" Oct 30 09:48:04.644993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:04.644960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:04.749417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:04.749371 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:04.845305 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:04.845263 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:05.532602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:05.532566 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:05.627141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:05.627104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:05.755578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:05.755542 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:05.854649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:05.854609 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:05.955225 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:05.955191 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:06.244946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:06.244838 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:06.366268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:06.366179 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:06.782718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:06.782636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:06.946723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:06.946686 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:08.612066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:08.612031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:09.062308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:09.062259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:09.169063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:09.169029 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:09.713304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:09.713269 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:10.372866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:10.372801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:10.463484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:10.463447 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:11.839023 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:11.838988 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:11.979989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:11.979873 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:12.073289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:12.073248 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:13.638183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:13.638147 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:13.738013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:13.737970 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:14.256241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:14.236454 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:15.871699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:15.871661 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:16.150617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:16.150534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:16.323719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:16.323684 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:16.416819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:16.416713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:17.568078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:17.568049 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:19.166294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:19.166259 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:19.495180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:19.495054 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:19.628240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:19.617911 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:20.844597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:20.844556 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:20.946708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:20.946660 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:21.044599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:21.044560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:21.148997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:21.148914 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:21.423033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:21.422941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:22.049068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:22.049032 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:22.146641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:22.146601 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:22.481294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:22.475702 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:22.597248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:22.582171 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:23.070506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:23.070462 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:23.843238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:23.843189 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:23.939279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:23.939206 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:25.103392 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:25.103346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:26.707914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:26.707877 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:27.044495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:27.044461 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:28.077537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:28.077506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:28.391833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:28.391748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:29.873419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:29.873385 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:29.996114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:29.996052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:30.342181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:30.342123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:31.137470 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:48:31.137427 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:48:31.954749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:31.954718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:32.618755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:32.618719 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:32.719492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:32.719459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:32.923950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:32.923855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:33.230817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:33.230727 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:33.448660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:33.448600 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:33.849284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:33.849242 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:33.949303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:33.949265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:35.352765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:35.352733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:35.475494 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:35.475455 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:36.306248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:36.306198 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:37.164390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:37.164351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:38.817759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:38.817671 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:39.291602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:39.291571 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:48:40.501138 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:40.501105 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:42.463895 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:42.463858 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:43.487619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:43.487577 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:48:43.612490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:43.612392 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 09:48:43.630352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:43.630295 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:48:43.851241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:43.850319 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:48:44.213297 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:44.207908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:45.745151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:45.745117 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:47.373121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:47.373092 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:48.955063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:48.955026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:49.832228 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:49.832175 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:50.701500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:50.701456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:52.163280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:52.163244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:53.241249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:53.241201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:54.329853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:54.329814 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:55.372120 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:55.372080 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:56.970200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:56.970160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:48:58.611322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:48:58.611274 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:00.290073 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:00.290041 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:00.568758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:00.568672 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:00.668788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:00.668754 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:01.443241 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:49:01.428792 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:49:02.358090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:02.358047 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:02.688323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:02.688241 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:03.198760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.198724 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:49:03.199934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.199903 4920 cpu_manager.go:414] "containerMap: removing container" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:49:03.199934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.199933 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.199971 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.199980 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efb2d2e-2f2b-4b2f-990e-83a91e0165f3" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200015 4920 cpu_manager.go:414] "containerMap: removing container" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200023 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0792860c-d44d-411d-b380-aeb7fcf9abe7" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200044 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200051 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa4f39-3541-40e5-8209-4863af3c0a73" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200078 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200085 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200102 4920 cpu_manager.go:414] "containerMap: removing container" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:49:03.200151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200109 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200281 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="49320e8c-dd17-474a-b5c0-8d2cec590723" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200305 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="622f3ba5-37a8-4777-b470-ca325878fbcf" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200322 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0b0a9d-b7b8-411a-8f3e-7987a73438c9" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200334 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ad24d6-6372-486e-b990-5a448f1ad0af" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200345 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea3ae92-781a-4cf7-b5f3-6c639fa04c2d" containerName="cleanup" Oct 30 09:49:03.200713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.200357 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a865623-785c-4e1b-a6df-3748ff6d140f" containerName="cleanup" Oct 30 09:49:03.204704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.204676 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.239595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.239565 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:49:03.246145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.246099 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f6ee2-openshift-default-dockercfg-lvhrd\"" Oct 30 09:49:03.280598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280538 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280598 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280590 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280618 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280650 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280695 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctml\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280755 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280787 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280864 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.280947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.280935 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.298743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.298713 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:03.390322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.390277 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.416828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.416795 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.417505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.417477 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.417781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.417763 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.418708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.418426 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.419022 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.418739 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.419505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.419486 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctml\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.426730 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.426663 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.426977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.426959 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.433239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.427764 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.433450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.433428 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.441496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.441473 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.441778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.441756 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.464060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.442831 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.464060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.463599 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.465040 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.465006 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.486237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.484041 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctml\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.486237 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.486044 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\") pod \"e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.523813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.523509 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:03.547910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.546639 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:03.754021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.753993 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:49:03.759280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.759159 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:49:03.976337 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:49:03.976308 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafe5514_75c3_4add_95bb_92b5759017a9.slice/crio-445fcd2f492bdea4df72901af37d259b365c2eb6efe0c72b941f4c9f0cfd99f1 WatchSource:0}: Error finding container 445fcd2f492bdea4df72901af37d259b365c2eb6efe0c72b941f4c9f0cfd99f1: Status 404 returned error can't find the container with id 445fcd2f492bdea4df72901af37d259b365c2eb6efe0c72b941f4c9f0cfd99f1 Oct 30 09:49:03.978646 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.978621 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:49:03.979754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.979718 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:49:03.979852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.979797 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:49:03.979852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:03.979839 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:49:04.247746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:04.247645 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerStarted","Data":"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b"} Oct 30 09:49:04.247746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:04.247720 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerStarted","Data":"445fcd2f492bdea4df72901af37d259b365c2eb6efe0c72b941f4c9f0cfd99f1"} Oct 30 09:49:04.300116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:04.300059 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podStartSLOduration=1.300041916 podStartE2EDuration="1.300041916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:49:04.29484998 +0000 UTC m=+3308.018277124" watchObservedRunningTime="2026-10-30 09:49:04.300041916 +0000 UTC m=+3308.023469061" Oct 30 09:49:04.341810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:04.341773 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:04.532700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:04.532324 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:05.538141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:05.538111 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:05.979000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:05.978910 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:06.262949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:06.262862 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:06.263902 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:06.263879 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:49:06.676694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:06.676655 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:06.772303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:06.772264 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:07.604758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:07.604721 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:08.343640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:08.343604 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:08.440992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:08.440957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:09.210130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:09.210095 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:10.757533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:10.757495 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:12.366682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:12.366646 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:13.850857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:13.850821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:15.375339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:15.375309 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:16.950580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:16.950531 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:18.423839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:18.423798 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:19.302153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:19.302115 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:49:19.906355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:19.906317 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:21.683811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:21.680915 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:22.865240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:22.864323 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 09:49:23.457082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:23.457039 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 09:49:23.527239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:23.526903 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:23.603258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:23.602623 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:49:23.682801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:23.682399 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:49:25.049350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:25.049304 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:26.534319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:26.534280 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:28.183656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:28.183612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:29.662202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:29.662153 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:31.139628 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:49:31.139580 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:49:31.220398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:31.220362 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:32.859320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:32.859270 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:34.432145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:34.432104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:36.011400 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:36.011355 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:37.563163 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:37.563068 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:39.243770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:39.243735 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:40.847936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:40.847902 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:42.361640 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:42.361602 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:43.962021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:43.961973 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:45.420454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:45.420405 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:46.842519 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.842480 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:49:46.847360 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.847337 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:46.866759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.866720 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:49:46.872992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.872955 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2f6ee2-serving-cert\"" Oct 30 09:49:46.910485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.910441 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:46.991856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.991823 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:46.992056 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.991947 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:46.992154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.992135 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:46.992247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.992229 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:46.993129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:46.993108 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:47.093493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.093407 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:47.093711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.093689 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.093899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.093881 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.093985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.093970 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.094089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.094073 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.094563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.094541 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.096345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.096325 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.096514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.096497 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.147246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.140998 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\") pod \"maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.172151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.168565 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:47.517136 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:49:47.517101 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c4c19f_4acd_4b1d_9212_58651e8d4a97.slice/crio-8bfa727c3fc3d324ea7afbec15cf973abdaca55bbb1277ab842eb7425d1f065a WatchSource:0}: Error finding container 8bfa727c3fc3d324ea7afbec15cf973abdaca55bbb1277ab842eb7425d1f065a: Status 404 returned error can't find the container with id 8bfa727c3fc3d324ea7afbec15cf973abdaca55bbb1277ab842eb7425d1f065a Oct 30 09:49:47.517972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.517849 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:49:47.519240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:47.519204 4920 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:49:48.013085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:48.013039 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" event={"ID":"01c4c19f-4acd-4b1d-9212-58651e8d4a97","Type":"ContainerStarted","Data":"8bfa727c3fc3d324ea7afbec15cf973abdaca55bbb1277ab842eb7425d1f065a"} Oct 30 09:49:48.546008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:48.545972 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:49.026322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:49.025244 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" event={"ID":"01c4c19f-4acd-4b1d-9212-58651e8d4a97","Type":"ContainerStarted","Data":"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365"} Oct 30 09:49:49.026322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:49.025308 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:49.068480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:49.068241 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" podStartSLOduration=2.5429665740000003 podStartE2EDuration="3.06820191s" totalImagesPullingTime="525.235336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:49:49.062794985 +0000 UTC m=+3352.786222152" watchObservedRunningTime="2026-10-30 09:49:49.06820191 +0000 UTC m=+3352.791629053" Oct 30 09:49:50.099933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:50.099900 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:51.569311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:51.569280 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:53.142055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:53.142020 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:53.376765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:53.376729 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:54.778536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:54.778497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:55.042509 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:55.042481 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:49:56.332506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:56.332465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:57.955160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:57.955123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:49:59.414402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:49:59.414358 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:00.968175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:00.968128 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:01.139093 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:50:01.139045 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:50:02.545568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:02.545534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:04.049423 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:04.049376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:05.493712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:05.493681 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:07.040848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:07.040801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:08.565000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:08.564917 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:10.057178 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:10.057136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:11.541175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:11.541134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:12.617242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:12.614002 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:50:13.157891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:13.157859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:14.872921 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:14.872882 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:16.460974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:16.460941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:17.932631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:17.932594 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:19.482044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:19.482007 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:20.962959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:20.962915 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:22.304395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:22.304351 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:50:22.304908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:22.304432 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:50:22.449205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:22.449135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:23.983451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:23.983412 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:24.385922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.385892 4920 generic.go:356] "Generic (PLEG): container finished" podID="aafe5514-75c3-4add-95bb-92b5759017a9" containerID="866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b" exitCode=0 Oct 30 09:50:24.386108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.385963 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerDied","Data":"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b"} Oct 30 09:50:24.386452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.386436 4920 scope.go:122] "RemoveContainer" containerID="866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b" Oct 30 09:50:24.388455 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.388426 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:24.388555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.388494 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:24.388555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:24.388522 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:25.396284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:25.396244 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerStarted","Data":"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a"} Oct 30 09:50:25.531384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:25.531341 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:50:25.531599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:25.531578 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:25.531652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:25.531628 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:25.645978 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:25.645945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:26.532942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:26.532901 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:26.533461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:26.532966 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:27.256606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:27.256575 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:27.531879 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:27.531767 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:27.532140 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:27.532106 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:28.531074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:28.531035 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:28.531655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:28.531107 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:28.836350 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:28.836304 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:29.532074 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:29.532029 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:29.532613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:29.532109 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:30.531887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:30.531849 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:30.532229 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:30.532172 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:30.549090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:30.549050 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:31.158758 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:50:31.158723 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:50:31.531254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:31.531148 4920 patch_prober.go:29] interesting pod/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 09:50:31.531416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:31.531252 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 09:50:32.071498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:32.071459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:32.536836 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:32.536802 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:50:32.537276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:32.537249 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:50:32.538027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:32.538005 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:50:33.563097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:33.563065 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:33.669835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:33.669801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:34.037322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.037292 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:34.500993 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.500869 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:50:34.508143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.508114 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.528123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.528092 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:50:34.534340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.534305 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bdd208-openshift-default-dockercfg-hzsdt\"" Oct 30 09:50:34.534530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.534498 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:34.651241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.630667 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.652406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.652365 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.652694 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.652672 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.652955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.652933 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.684240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.656322 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.684558 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.684524 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.715401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.715357 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gpk\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.715783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.715745 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.716005 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.715980 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824443 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824508 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824632 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824695 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824759 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824840 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824906 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824944 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.824994 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gpk\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.826877 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.827972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.827585 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.828396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.828372 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.839284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.828807 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.839497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.834416 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.870323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.853506 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.893236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.887839 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gpk\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.904241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.896351 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:34.908239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.907109 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:34.908239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:34.907896 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:35.129935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.129858 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:35.262835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.262786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:35.450553 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:50:35.450521 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccea84a3_36c1_4d20_a018_59db9ed75f16.slice/crio-fe1c9b285d193e101b90f3a8e800580d5349b23cd109421bb5129d6fe562ee82 WatchSource:0}: Error finding container fe1c9b285d193e101b90f3a8e800580d5349b23cd109421bb5129d6fe562ee82: Status 404 returned error can't find the container with id fe1c9b285d193e101b90f3a8e800580d5349b23cd109421bb5129d6fe562ee82 Oct 30 09:50:35.453190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.453165 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:50:35.453700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.453672 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:35.453753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.453732 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:35.453793 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.453763 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:50:35.535943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:35.535908 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerStarted","Data":"fe1c9b285d193e101b90f3a8e800580d5349b23cd109421bb5129d6fe562ee82"} Oct 30 09:50:36.546045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:36.546001 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerStarted","Data":"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7"} Oct 30 09:50:36.600565 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:36.600506 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podStartSLOduration=2.600490734 podStartE2EDuration="2.600490734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:50:36.597284909 +0000 UTC m=+3400.320712073" watchObservedRunningTime="2026-10-30 09:50:36.600490734 +0000 UTC m=+3400.323917876" Oct 30 09:50:36.890702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:36.890618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:37.130593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:37.130554 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:37.137437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:37.137403 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:37.554209 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:37.554176 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:37.555302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:37.555279 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:50:38.561321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:38.561210 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:38.963054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:38.962930 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:39.088673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:39.088621 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:40.258692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:40.258659 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:40.689192 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:40.689157 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:40.789866 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:40.789833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:41.953727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:41.953686 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:43.463324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:43.463287 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:44.969039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:44.968998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:46.455262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:46.455195 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:47.951345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:47.951299 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:49.545481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:49.545437 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:51.099932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:51.099892 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:52.664299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:52.664261 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:53.498970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:53.498933 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:54.177499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:54.177456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:55.740175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:55.740136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:57.379132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:57.379086 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:50:58.932165 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:50:58.932126 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:00.428697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:00.428656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:00.547238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:00.540299 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:00.857593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:00.857509 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:01.229407 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:51:01.221482 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:51:02.446323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.446279 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:02.548319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.547981 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:02.858711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.858671 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:51:02.864411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.864357 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.895970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.895939 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:51:02.902898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.902871 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82ccfd-openshift-default-dockercfg-rlbdn\"" Oct 30 09:51:02.904883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.904857 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:02.987131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987094 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrvx\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987238 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987298 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987619 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987680 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987763 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987799 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987863 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:02.987924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:02.987905 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.002806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.002753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:03.091735 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.091693 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrvx\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.092034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.092013 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.092179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.092163 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.097744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.097712 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.104478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.104449 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.104679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.104660 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.104805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.104791 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.121288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.104917 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.121609 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.121578 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.122014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.092980 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.122246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.106090 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.138235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.129454 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.138428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.138327 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.142971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.139244 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.178242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.177889 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.200800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.200764 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.201783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.201757 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrvx\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.211240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.207237 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.215277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.211728 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:03.707896 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:51:03.707862 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9c5ac0_342e_4fca_838f_f2871af96e55.slice/crio-cda26b901cdbfa068e8c1818d5f8ca6bcf791cc5574d418b4025788d488f2d69 WatchSource:0}: Error finding container cda26b901cdbfa068e8c1818d5f8ca6bcf791cc5574d418b4025788d488f2d69: Status 404 returned error can't find the container with id cda26b901cdbfa068e8c1818d5f8ca6bcf791cc5574d418b4025788d488f2d69 Oct 30 09:51:03.709197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.709171 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:51:03.711490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.711452 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:51:03.711579 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.711539 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:51:03.711626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.711575 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:51:03.864084 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.864044 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerStarted","Data":"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956"} Oct 30 09:51:03.864332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.864122 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerStarted","Data":"cda26b901cdbfa068e8c1818d5f8ca6bcf791cc5574d418b4025788d488f2d69"} Oct 30 09:51:03.905264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:03.905195 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podStartSLOduration=1.905180692 podStartE2EDuration="1.905180692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:03.903590328 +0000 UTC m=+3427.627017494" watchObservedRunningTime="2026-10-30 09:51:03.905180692 +0000 UTC m=+3427.628607835" Oct 30 09:51:04.173326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:04.173291 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:04.213362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:04.213260 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:04.214965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:04.214941 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:51:04.215096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:04.214986 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:51:05.220105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:05.220069 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:05.874583 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:05.874534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:05.882061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:05.882033 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:05.882882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:05.882858 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:51:06.268669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:06.268579 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:06.385700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:06.385663 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:07.669079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:07.669046 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:08.076843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:08.076805 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:08.187727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:08.187692 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:09.489928 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:09.489889 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:10.970325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:10.970292 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:12.573924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:12.573888 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:14.118288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:14.118201 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:51:14.169877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:14.169842 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:15.874198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:15.874155 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:17.849006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:17.848955 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n" podUID="46e36101-1742-43d6-82dd-010f3d70bcf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:51:17.858452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:17.858403 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:17.913985 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:17.913928 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:51:18.041362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:18.039405 4920 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:51:18.510242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:18.488960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:18.711685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:18.703439 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:19.246242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:19.241331 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:19.472240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:19.468811 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:51:19.533740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:19.507277 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:51:19.535547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:19.530311 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 09:51:20.065361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:20.065321 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:21.671338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:21.671301 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:23.043553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.043519 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:51:23.048152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.048134 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.074654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.074623 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bdd208-serving-cert\"" Oct 30 09:51:23.081285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.081256 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:51:23.109415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.109358 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.109719 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.109694 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkpf2\" (UniqueName: \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.109911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.109892 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.110057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.110041 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.117355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.117331 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:23.212204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.212167 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.212425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.212245 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkpf2\" (UniqueName: \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.212425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.212284 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.212425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.212330 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.212997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.212968 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.215034 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.215013 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.215183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.215161 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.217746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.217722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:23.231597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.231572 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkpf2\" (UniqueName: \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\") pod \"maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.328496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.323058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:23.363817 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.363782 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:23.674106 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:51:23.674074 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9de14d_2ed8_4644_806f_f034ed71b659.slice/crio-d22024d6078b66f996380914d874ba372383a90e12261b0da174ac2b746805b1 WatchSource:0}: Error finding container d22024d6078b66f996380914d874ba372383a90e12261b0da174ac2b746805b1: Status 404 returned error can't find the container with id d22024d6078b66f996380914d874ba372383a90e12261b0da174ac2b746805b1 Oct 30 09:51:23.676555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:23.676441 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:51:24.475704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.475664 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" event={"ID":"7b9de14d-2ed8-4644-806f-f034ed71b659","Type":"ContainerStarted","Data":"77822ef47a6ca338b3c7943efafac762ec40a8d84ad62871c63fe1a5cdb8f6b7"} Oct 30 09:51:24.476105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.475729 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" event={"ID":"7b9de14d-2ed8-4644-806f-f034ed71b659","Type":"ContainerStarted","Data":"d22024d6078b66f996380914d874ba372383a90e12261b0da174ac2b746805b1"} Oct 30 09:51:24.476105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.475757 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:24.517484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.517421 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" podStartSLOduration=1.108521766 podStartE2EDuration="1.517401037s" totalImagesPullingTime="408.879271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:24.514827444 +0000 UTC m=+3448.238254606" watchObservedRunningTime="2026-10-30 09:51:24.517401037 +0000 UTC m=+3448.240828184" Oct 30 09:51:24.797716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.797673 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:24.898134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:24.898094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:26.616197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:26.616159 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:28.249476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:28.249433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:29.795002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:29.794963 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:30.516972 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:30.516937 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:51:31.136501 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:51:31.136457 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:51:31.287373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:31.287334 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:32.862648 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:32.862587 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:34.350045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:34.350009 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:35.907760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:35.907727 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:37.429080 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:37.429047 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:38.550760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:38.550649 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:38.631722 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:38.631685 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:39.067208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:39.067176 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:40.524975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:40.524936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:41.985275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:41.985206 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:43.474288 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:43.474256 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:45.035414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:45.035372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:46.591934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:46.591886 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:47.904761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:47.904670 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:48.074664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:48.074621 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:49.624915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:49.624878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:51.051977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:51.051941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:51.136780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:51.136749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:51.249428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:51.249391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:52.765130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:52.765094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:53.617713 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:53.617678 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:54.392285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:54.392250 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:56.057046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:56.057008 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:57.632171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:57.632132 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:51:59.126623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:51:59.126587 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:00.642090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:00.642054 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:01.156104 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:52:01.156067 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:52:02.147449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:02.147415 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:03.638295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:03.638251 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:05.263055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:05.263016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:06.091055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.091023 4920 generic.go:356] "Generic (PLEG): container finished" podID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerID="9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7" exitCode=0 Oct 30 09:52:06.091259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.091087 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerDied","Data":"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7"} Oct 30 09:52:06.091586 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.091567 4920 scope.go:122] "RemoveContainer" containerID="9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7" Oct 30 09:52:06.093622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.093590 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:06.093696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.093662 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:06.093733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.093696 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:06.743698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:06.743622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:07.137033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:07.136986 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerStarted","Data":"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c"} Oct 30 09:52:07.377046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:07.377010 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:07.477799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:07.477706 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:07.554643 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:07.554612 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:52:08.130142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:08.130107 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:52:08.131858 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:08.131828 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 09:52:08.132001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:08.131882 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 09:52:08.185906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:08.185860 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:09.130363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:09.130327 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 09:52:09.130893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:09.130389 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 09:52:09.795236 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:09.784961 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:10.130671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:10.130585 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 09:52:10.130671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:10.130640 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 09:52:11.042890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:11.042850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:11.131718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:11.131682 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 09:52:11.132329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:11.132292 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 09:52:11.372950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:11.372858 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:12.130452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:12.130377 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 09:52:12.130452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:12.130438 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 09:52:12.937620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:12.937349 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:13.126800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.126766 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:52:13.137823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.137797 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.149522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.149470 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:52:13.154891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.154870 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f6ee2\"/\"default-dockercfg-v8m78\"" Oct 30 09:52:13.155290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.155273 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f6ee2\"/\"test-model-2f6ee2-kserve-self-signed-certs\"" Oct 30 09:52:13.164141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.164019 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f6ee2\"/\"openshift-service-ca.crt\"" Oct 30 09:52:13.164141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.164105 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f6ee2\"/\"kube-root-ca.crt\"" Oct 30 09:52:13.168438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.168397 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:13.210450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.210368 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:52:13.213617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.213370 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:52:13.334266 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334180 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.334470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334315 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.334470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334353 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.334592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334517 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.334592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334549 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slht\" (UniqueName: \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.334742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.334713 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.439332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.439285 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.444242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.440501 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.456375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.444840 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.470813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.470703 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.471278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.471255 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.471943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.471916 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slht\" (UniqueName: \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.569795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.471838 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.569795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.565885 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.569795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.566763 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.586319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.585656 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.586319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.586293 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.589850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.589818 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slht\" (UniqueName: \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\") pod \"test-model-2f6ee2-kserve-796dd788b7-88x6j\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:13.756548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:13.756454 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:14.065329 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:52:14.065285 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61b96941_be95_4b6c_bd41_8d8cef2f4fd6.slice/crio-48c4fce800c29257223cd7a2c3ca6246861ce61f8515ad3218a98d6ad77cece5 WatchSource:0}: Error finding container 48c4fce800c29257223cd7a2c3ca6246861ce61f8515ad3218a98d6ad77cece5: Status 404 returned error can't find the container with id 48c4fce800c29257223cd7a2c3ca6246861ce61f8515ad3218a98d6ad77cece5 Oct 30 09:52:14.066898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.066869 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:52:14.199355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.198881 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" event={"ID":"61b96941-be95-4b6c-bd41-8d8cef2f4fd6","Type":"ContainerStarted","Data":"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee"} Oct 30 09:52:14.199355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.198939 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" event={"ID":"61b96941-be95-4b6c-bd41-8d8cef2f4fd6","Type":"ContainerStarted","Data":"48c4fce800c29257223cd7a2c3ca6246861ce61f8515ad3218a98d6ad77cece5"} Oct 30 09:52:14.237886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.237824 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" podStartSLOduration=1.23780494 podStartE2EDuration="1.23780494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:14.235548377 +0000 UTC m=+3497.958975520" watchObservedRunningTime="2026-10-30 09:52:14.23780494 +0000 UTC m=+3497.961232085" Oct 30 09:52:14.591500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.591463 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:14.757025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:14.756988 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:16.144262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:16.144225 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:17.702651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:17.702612 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:19.339881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:19.339846 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:20.945926 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:20.945888 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:22.402943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:22.402904 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:23.757472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:23.757428 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:23.770618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:23.770591 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:23.940043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:23.940010 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:24.355141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:24.355098 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:52:25.656966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:25.656932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:27.195825 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:27.195792 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:28.794987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:28.794948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:30.357727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:30.357691 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:30.491763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:30.491722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:31.137384 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:52:31.137347 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:52:31.822347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:31.822306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:32.343042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:32.343003 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:52:33.231316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:33.231277 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:33.615485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:33.615448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:35.217201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:35.217168 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:36.800160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:36.800119 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:38.533773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:38.533733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:40.062748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:40.062699 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:41.613827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:41.613725 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:43.158990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:43.158938 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:43.407001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:43.406944 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:43.857155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:43.857124 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:44.262075 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.262035 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:52:44.269541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.269510 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.299740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.299706 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a479df-openshift-default-dockercfg-fjqpv\"" Oct 30 09:52:44.328081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.327991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:44.334488 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.334452 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347370 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzbc\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347479 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347634 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347701 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347922 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.347957 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.348240 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.348303 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.350901 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.348532 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.570822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.570784 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.689937 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.689894 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.691007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.690967 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.693351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.575894 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.698518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.697924 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.698518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.697986 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.698518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.698067 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.698518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.698124 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.698518 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.698159 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.700795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.698897 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzbc\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.700795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.699579 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.700795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.699785 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.700795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.699875 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.704637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.702936 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.704637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.703520 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.745170 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.745048 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:44.749486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.748915 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.751968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.751878 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzbc\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.770442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.766338 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\") pod \"e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.911155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.910635 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:44.913247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:44.913189 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:45.266891 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:52:45.266854 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc96b8ff_00f2_486f_9eaa_916e9695fdd7.slice/crio-269297352fd0af3cdcc4c93aa779a00b5ef314766e6d1da635f0a1a7daf9ac6d WatchSource:0}: Error finding container 269297352fd0af3cdcc4c93aa779a00b5ef314766e6d1da635f0a1a7daf9ac6d: Status 404 returned error can't find the container with id 269297352fd0af3cdcc4c93aa779a00b5ef314766e6d1da635f0a1a7daf9ac6d Oct 30 09:52:45.268372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.268347 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:52:45.270727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.270693 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:45.270833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.270777 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:45.270833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.270811 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:45.889480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.889441 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerStarted","Data":"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4"} Oct 30 09:52:45.889661 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.889502 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerStarted","Data":"269297352fd0af3cdcc4c93aa779a00b5ef314766e6d1da635f0a1a7daf9ac6d"} Oct 30 09:52:45.911375 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.911344 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:45.947197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:45.947145 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podStartSLOduration=1.947130847 podStartE2EDuration="1.947130847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:45.942658591 +0000 UTC m=+3529.666085737" watchObservedRunningTime="2026-10-30 09:52:45.947130847 +0000 UTC m=+3529.670557990" Oct 30 09:52:46.471211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:46.471179 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:46.920512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:46.920475 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:47.910300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:47.910261 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:47.911551 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:47.911527 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:52:48.027193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:48.024984 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:48.401130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:48.401095 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:48.503554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:48.503515 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:49.777151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.777113 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:49.936541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.936504 4920 generic.go:356] "Generic (PLEG): container finished" podID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerID="eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956" exitCode=0 Oct 30 09:52:49.936715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.936583 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerDied","Data":"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956"} Oct 30 09:52:49.937251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.937209 4920 scope.go:122] "RemoveContainer" containerID="eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956" Oct 30 09:52:49.940017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.939981 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:49.940290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.940259 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:49.940463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:49.940444 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:50.061728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:50.061672 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:50.216312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:50.216271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:50.343471 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:50.343346 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:50.883031 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:50.882992 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:52:50.952933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:50.952891 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerStarted","Data":"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102"} Oct 30 09:52:51.213372 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:51.213268 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:52:51.214552 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:51.214526 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:51.214698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:51.214578 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:51.563938 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:51.563898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:52.218751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:52.218699 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:52.220450 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:52.220414 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:53.214665 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:53.214623 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:53.214987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:53.214948 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:53.391021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:53.390983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:53.746856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:53.746824 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:54.222238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:54.221636 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:54.222238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:54.221702 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:55.197181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:55.197150 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:55.215241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:55.214627 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:55.215241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:55.214704 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:56.215668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:56.215627 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 09:52:56.216374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:56.216338 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 09:52:56.846177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:56.846129 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:57.219427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:57.219343 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:52:57.219816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:57.219734 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:52:57.220427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:57.220405 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:52:58.333673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:58.333636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:59.443490 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:59.443446 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:52:59.565311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:52:59.565270 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:00.166007 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:00.165973 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:00.265535 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:00.265497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:01.140958 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:53:01.140917 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:53:01.202677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:01.202626 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:01.302079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:01.302040 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:01.500634 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:01.500547 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:01.828268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:01.823864 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:02.085292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:02.085167 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:02.451532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:02.451456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:02.552070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:02.552038 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:04.283089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:04.283052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:04.765775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:04.765736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:05.735662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:05.735630 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:06.212240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:06.205960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:07.901101 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:07.901056 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:09.383935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:09.383895 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:09.747368 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:09.747132 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:09.835417 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:09.835375 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:10.362411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:10.362375 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:10.665429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:10.665345 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:11.326427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:11.326394 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:12.802925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:12.802884 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:14.375401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:14.375366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:15.968828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:15.968789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:16.514697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:16.514666 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:16.637172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:16.637129 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:17.813045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:17.813005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:19.448208 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:19.448171 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:20.984416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:20.984376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:22.720504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:22.720459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:24.377401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:24.377366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:25.929787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:25.929747 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:26.671859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:26.671485 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:26.772246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:26.772182 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:27.987394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:27.987357 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:28.152892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:28.152840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:29.486546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:29.477105 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:30.049175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:30.049144 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:30.594049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:30.594009 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:30.692893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:30.692855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:31.400404 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:53:31.400366 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:53:31.739970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:31.739873 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:33.337388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.337350 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:53:33.343319 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.343294 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.360181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.360150 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:53:33.360980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.360947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:33.487597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487556 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487607 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487636 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487801 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487776 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487808 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487846 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kk2\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487878 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.487995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.487950 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.488202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.488045 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.589827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.589731 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.589827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.589783 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.589827 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.589819 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kk2\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.589849 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.589939 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590061 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590360 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590384 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590433 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590460 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590595 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590842 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.590956 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.590907 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.591128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.591107 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.592576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.592552 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.593115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.593094 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.610154 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.610121 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.610352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.610202 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kk2\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.663534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.663497 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:33.996276 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:53:33.996238 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3528fc4f_2bf6_499b_965a_a1738c881609.slice/crio-dc35a2fac055db559a67fa08173976d9463e4b7d8407ea55c91d7107211d00a2 WatchSource:0}: Error finding container dc35a2fac055db559a67fa08173976d9463e4b7d8407ea55c91d7107211d00a2: Status 404 returned error can't find the container with id dc35a2fac055db559a67fa08173976d9463e4b7d8407ea55c91d7107211d00a2 Oct 30 09:53:33.999336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.999306 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:53:33.999688 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.999655 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:33.999760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.999727 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:33.999818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:33.999758 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:34.792156 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:34.792114 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" event={"ID":"3528fc4f-2bf6-499b-965a-a1738c881609","Type":"ContainerStarted","Data":"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6"} Oct 30 09:53:34.792613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:34.792170 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" event={"ID":"3528fc4f-2bf6-499b-965a-a1738c881609","Type":"ContainerStarted","Data":"dc35a2fac055db559a67fa08173976d9463e4b7d8407ea55c91d7107211d00a2"} Oct 30 09:53:34.857614 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:34.857551 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podStartSLOduration=1.857535121 podStartE2EDuration="1.857535121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:34.842658895 +0000 UTC m=+3578.566086065" watchObservedRunningTime="2026-10-30 09:53:34.857535121 +0000 UTC m=+3578.580962305" Oct 30 09:53:34.965831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:34.965797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:35.664082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:35.664046 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:35.666171 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:35.666145 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:35.666310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:35.666196 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:36.443461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:36.443316 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:36.664856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:36.664820 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:36.665046 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:36.664876 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:37.664315 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.664278 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:37.664746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.664340 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:37.843789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.843749 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:53:37.849767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.849741 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.866889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.866854 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a479df-serving-cert\"" Oct 30 09:53:37.872599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.872569 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:53:37.874049 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.874022 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.874491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.874467 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.874850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.874626 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.874850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.874715 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjjp\" (UniqueName: \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.951320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.940636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:37.976698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.976657 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.977473 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.977430 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.977626 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.977527 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjjp\" (UniqueName: \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.977701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.977625 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.978499 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:53:37.978472 4920 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-serving-cert: secret "maas-api-e2e-inf-b-a479df-serving-cert" not found Oct 30 09:53:37.978615 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:53:37.978547 4920 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls podName:f2eebedf-163f-4cd0-9c4f-61483f191427 nodeName:}" failed. No retries permitted until 2026-10-30 09:53:38.478526322 +0000 UTC m=+3582.201953445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls") pod "maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" (UID: "f2eebedf-163f-4cd0-9c4f-61483f191427") : secret "maas-api-e2e-inf-b-a479df-serving-cert" not found Oct 30 09:53:37.978729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.978703 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:37.982270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:37.982188 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:38.106253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.102855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:38.129778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.110020 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjjp\" (UniqueName: \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:38.247805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.247658 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:38.524276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.524151 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:38.535762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.535725 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") pod \"maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:38.665250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.664605 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:38.665250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.664665 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:38.789739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:38.789706 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:39.358994 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:53:39.358965 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2eebedf_163f_4cd0_9c4f_61483f191427.slice/crio-271c23a9311851fb48d10a3d5bdcdb2981941f6ad4ab22a88a78b6ca2d9ca77f WatchSource:0}: Error finding container 271c23a9311851fb48d10a3d5bdcdb2981941f6ad4ab22a88a78b6ca2d9ca77f: Status 404 returned error can't find the container with id 271c23a9311851fb48d10a3d5bdcdb2981941f6ad4ab22a88a78b6ca2d9ca77f Oct 30 09:53:39.364284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:39.364258 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:53:39.644591 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:39.644512 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:39.664880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:39.664850 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:39.665057 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:39.664897 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:39.883960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:39.883920 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" event={"ID":"f2eebedf-163f-4cd0-9c4f-61483f191427","Type":"ContainerStarted","Data":"271c23a9311851fb48d10a3d5bdcdb2981941f6ad4ab22a88a78b6ca2d9ca77f"} Oct 30 09:53:40.664553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:40.664510 4920 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 30 09:53:40.664751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:40.664578 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 30 09:53:40.894458 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:40.894424 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" event={"ID":"f2eebedf-163f-4cd0-9c4f-61483f191427","Type":"ContainerStarted","Data":"815cbf4fb0959f9e2dba81b93dfb60d4b6006f28fdd7fc210b18f089897219c2"} Oct 30 09:53:40.894877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:40.894648 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:40.935567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:40.935459 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" podStartSLOduration=3.335210023 podStartE2EDuration="3.935445509s" totalImagesPullingTime="600.235486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:40.932332221 +0000 UTC m=+3584.655759383" watchObservedRunningTime="2026-10-30 09:53:40.935445509 +0000 UTC m=+3584.658872651" Oct 30 09:53:41.119029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:41.118987 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:41.669894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:41.669855 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:41.670406 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:41.670381 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:41.671280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:41.671257 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:53:42.380395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:42.377902 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:53:42.381079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:42.380525 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" containerName="maas-api" containerID="cri-o://6973ff521114a331ce5bd7d3cd98a8920e147ad73ca9479f62690aae40b8e8d0" gracePeriod=30 Oct 30 09:53:42.424676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:42.424638 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:42.487334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:42.487291 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.243:8443/health\": dial tcp 10.132.0.243:8443: connect: connection refused" Oct 30 09:53:43.113242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.110894 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:43.169242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.168339 4920 generic.go:356] "Generic (PLEG): container finished" podID="ac433f30-005f-4f0e-94f3-779fb966312d" containerID="6973ff521114a331ce5bd7d3cd98a8920e147ad73ca9479f62690aae40b8e8d0" exitCode=0 Oct 30 09:53:43.181324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.173483 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" event={"ID":"ac433f30-005f-4f0e-94f3-779fb966312d","Type":"ContainerDied","Data":"6973ff521114a331ce5bd7d3cd98a8920e147ad73ca9479f62690aae40b8e8d0"} Oct 30 09:53:43.196982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.195975 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:43.223607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.223573 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:53:43.296791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.296759 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:43.309028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.308994 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\") pod \"ac433f30-005f-4f0e-94f3-779fb966312d\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " Oct 30 09:53:43.309200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.309132 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\" (UniqueName: \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\") pod \"ac433f30-005f-4f0e-94f3-779fb966312d\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " Oct 30 09:53:43.309200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.309186 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\") pod \"ac433f30-005f-4f0e-94f3-779fb966312d\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " Oct 30 09:53:43.309412 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.309312 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\") pod \"ac433f30-005f-4f0e-94f3-779fb966312d\" (UID: \"ac433f30-005f-4f0e-94f3-779fb966312d\") " Oct 30 09:53:43.309477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.309417 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca" pod "ac433f30-005f-4f0e-94f3-779fb966312d" (UID: "ac433f30-005f-4f0e-94f3-779fb966312d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:43.310212 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.310185 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac433f30-005f-4f0e-94f3-779fb966312d-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.312624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.312591 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls" pod "ac433f30-005f-4f0e-94f3-779fb966312d" (UID: "ac433f30-005f-4f0e-94f3-779fb966312d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:43.314479 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.314451 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r" pod "ac433f30-005f-4f0e-94f3-779fb966312d" (UID: "ac433f30-005f-4f0e-94f3-779fb966312d"). InnerVolumeSpecName "kube-api-access-lpz8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:43.315644 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.315609 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs" pod "ac433f30-005f-4f0e-94f3-779fb966312d" (UID: "ac433f30-005f-4f0e-94f3-779fb966312d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:43.412002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.411901 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpz8r\" (UniqueName: \"kubernetes.io/projected/ac433f30-005f-4f0e-94f3-779fb966312d-kube-api-access-lpz8r\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.412595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.412572 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.412698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.412687 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac433f30-005f-4f0e-94f3-779fb966312d-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:53:43.717461 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:43.717361 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:44.178527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.178488 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" Oct 30 09:53:44.178527 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.178507 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s" event={"ID":"ac433f30-005f-4f0e-94f3-779fb966312d","Type":"ContainerDied","Data":"1858988c232004fee1cba4520c1263bf720e1b03d1c536a92053bc5d63448dba"} Oct 30 09:53:44.178816 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.178569 4920 scope.go:122] "RemoveContainer" containerID="6973ff521114a331ce5bd7d3cd98a8920e147ad73ca9479f62690aae40b8e8d0" Oct 30 09:53:44.231555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.231514 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:53:44.246115 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.246075 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-881f83-794dbfc9b5-gs27s"] Oct 30 09:53:44.401337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:44.401290 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" path="/var/lib/kubelet/pods/ac433f30-005f-4f0e-94f3-779fb966312d/volumes" Oct 30 09:53:45.000607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:45.000573 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:45.893727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:45.893689 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:53:46.575892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:46.575856 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:46.974444 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:46.965536 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:53:47.930953 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:47.930893 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:53:48.337301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:48.337265 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:49.133706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:49.133663 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z896s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:53:49.134362 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:49.134329 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z896s" podUID="d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:53:50.148241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.144875 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:50.296116 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.296058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:50.435193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.435106 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:50.815763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.815727 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:50.926235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.925656 4920 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-knmrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:53:50.926235 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:50.925736 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:53:51.281955 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:51.281865 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:51.398880 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:51.398841 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:51.591782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:51.591755 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:52.579030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:52.578990 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:52.679342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:52.679305 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:52.763205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:52.763166 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:52.851082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:52.851001 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:53.229300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:53.222262 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:53.734621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:53.734586 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:53.883887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:53.883849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:54.545387 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:54.545334 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:56.335981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:56.335945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:57.008946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:57.008909 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:57.121539 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:57.121492 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:57.855522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:57.855480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:59.146141 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:59.146100 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:59.248577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:59.248543 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:53:59.399823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:53:59.399738 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:01.142525 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:54:01.142484 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:54:01.208374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:01.208338 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:02.691893 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:02.691859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:02.883259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:02.883182 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:02.968276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:02.968135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:03.832206 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:03.832173 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:54:03.836210 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:03.836188 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:54:04.288892 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:04.288856 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:05.870131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:05.870072 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:07.441516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:07.441482 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:07.860105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:07.860068 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:08.014242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:08.010779 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:09.021397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:09.021360 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:09.987476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:09.987433 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:10.120239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:10.119532 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:10.616849 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:10.616714 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:12.272783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:12.272736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:13.848732 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:13.848695 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:15.341285 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:15.341246 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:16.905602 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:16.905563 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:18.442532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:18.442500 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:20.079259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:20.079180 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:21.565797 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:21.565763 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:22.054649 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:22.054617 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:23.034117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:23.034079 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:24.588172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:24.588130 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:26.172811 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:26.172772 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:27.729725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:27.729688 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:29.510767 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:29.510682 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:30.739241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:30.660205 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:31.396607 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:54:31.396561 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:54:31.448936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:31.448898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:32.270914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:32.270867 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:33.165070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:33.165028 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:33.763963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:33.763920 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:34.613884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:34.613840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:36.293642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:36.293588 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:37.848982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:37.848924 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:39.100812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:39.100468 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:39.713200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:39.713134 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:39.821486 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:39.821449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:40.181881 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:40.181849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:41.385748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.385701 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:41.792547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.792514 4920 generic.go:356] "Generic (PLEG): container finished" podID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerID="9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4" exitCode=0 Oct 30 09:54:41.792769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.792595 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerDied","Data":"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4"} Oct 30 09:54:41.793081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.793066 4920 scope.go:122] "RemoveContainer" containerID="9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4" Oct 30 09:54:41.795087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.795060 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:54:41.795194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.795143 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:54:41.795194 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:41.795177 4920 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:54:42.750051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:42.750016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:42.804047 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:42.804015 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerStarted","Data":"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad"} Oct 30 09:54:42.921239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:42.920618 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:54:42.930243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:42.926343 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:42.930243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:42.926447 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:43.340241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.335281 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:43.674291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.670664 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:43.765161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.765123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:43.918728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.918689 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:43.919045 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.919013 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:43.984842 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:43.984760 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:44.914776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:44.914727 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:44.915919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:44.915885 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:45.175528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:45.175432 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:45.916124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:45.915674 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:45.916124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:45.915737 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:46.227962 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.227832 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:46.327707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.327668 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:46.544998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.544947 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:46.552812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.552781 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:54:46.553083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.553056 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" containerID="cri-o://9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279" gracePeriod=30 Oct 30 09:54:46.712172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.712041 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:46.916667 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.916566 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:46.917310 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:46.917275 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:47.910343 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:47.910308 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:54:47.915239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:47.912315 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:47.915239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:47.912374 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:48.143745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.143700 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:54:48.144320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.143899 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerName="maas-api" containerID="cri-o://4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b" gracePeriod=30 Oct 30 09:54:48.181137 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.181094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:48.285144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.285096 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.251:8443/health\": dial tcp 10.132.0.251:8443: connect: connection refused" Oct 30 09:54:48.840239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.839523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:48.919398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.919364 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 09:54:48.935284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.931388 4920 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 09:54:48.986093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:48.982998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:49.004260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.003139 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:54:49.010252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.005739 4920 generic.go:356] "Generic (PLEG): container finished" podID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerID="4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b" exitCode=0 Oct 30 09:54:49.010252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.005959 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" event={"ID":"66ce10d9-d408-497c-9b17-42ec3433d78a","Type":"ContainerDied","Data":"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b"} Oct 30 09:54:49.010252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.006016 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" event={"ID":"66ce10d9-d408-497c-9b17-42ec3433d78a","Type":"ContainerDied","Data":"a712d1dbcad4c29dc3dbda5ffd16698125c293d1f29dfe8204beffe2bc16447f"} Oct 30 09:54:49.010252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.006044 4920 scope.go:122] "RemoveContainer" containerID="4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b" Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.139203 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\") pod \"66ce10d9-d408-497c-9b17-42ec3433d78a\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.139311 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\" (UniqueName: \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\") pod \"66ce10d9-d408-497c-9b17-42ec3433d78a\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.139523 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\") pod \"66ce10d9-d408-497c-9b17-42ec3433d78a\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.139624 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\") pod \"66ce10d9-d408-497c-9b17-42ec3433d78a\" (UID: \"66ce10d9-d408-497c-9b17-42ec3433d78a\") " Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.140643 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca" pod "66ce10d9-d408-497c-9b17-42ec3433d78a" (UID: "66ce10d9-d408-497c-9b17-42ec3433d78a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.145493 4920 scope.go:122] "RemoveContainer" containerID="4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b" Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:54:49.146358 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b\": container with ID starting with 4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b not found: ID does not exist" containerID="4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b" Oct 30 09:54:49.147203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.146392 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b"} err="failed to get container status \"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b\": rpc error: code = NotFound desc = could not find container \"4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b\": container with ID starting with 4a19f05fbe32d0bc6ae2a9c862f685faf5be763284f09d096c185e10d40c531b not found: ID does not exist" Oct 30 09:54:49.188280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.186351 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md" pod "66ce10d9-d408-497c-9b17-42ec3433d78a" (UID: "66ce10d9-d408-497c-9b17-42ec3433d78a"). InnerVolumeSpecName "kube-api-access-vk5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:49.194076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.193412 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls" pod "66ce10d9-d408-497c-9b17-42ec3433d78a" (UID: "66ce10d9-d408-497c-9b17-42ec3433d78a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:49.194076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.193906 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:49.202430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.202382 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs" pod "66ce10d9-d408-497c-9b17-42ec3433d78a" (UID: "66ce10d9-d408-497c-9b17-42ec3433d78a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:54:49.250890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.250780 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:49.250890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.250816 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ce10d9-d408-497c-9b17-42ec3433d78a-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:49.250890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.250831 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ce10d9-d408-497c-9b17-42ec3433d78a-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:49.250890 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.250844 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5md\" (UniqueName: \"kubernetes.io/projected/66ce10d9-d408-497c-9b17-42ec3433d78a-kube-api-access-vk5md\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:49.444948 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.444859 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:49.917546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.917509 4920 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:54:49.918854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:49.918824 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:54:50.016660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.016630 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9" Oct 30 09:54:50.087457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.087420 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:54:50.091697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.091669 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88cbc7-77f749cb76-69xh9"] Oct 30 09:54:50.290653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.290623 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:50.385241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.384855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:50.426293 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.423633 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" path="/var/lib/kubelet/pods/66ce10d9-d408-497c-9b17-42ec3433d78a/volumes" Oct 30 09:54:50.613348 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.613102 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:54:50.613548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.613402 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" containerID="cri-o://2eb7935079e2f2a28bce0f8739dec61d14ed2b792be6d13ad7a82e2b435350e5" gracePeriod=30 Oct 30 09:54:50.624487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.624456 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:50.756283 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:50.752414 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:51.845233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.845200 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:54:51.912066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912033 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912117 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912263 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912337 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912477 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912410 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912475 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.912534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912485 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:51.912534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912511 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.913701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912548 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.913701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912662 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\") pod \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\" (UID: \"054f2f7e-0462-4f88-8cf1-56888d1d20d0\") " Oct 30 09:54:51.913701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.912993 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:51.913701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.913294 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:51.913701 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.913541 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:51.913986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.913715 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:51.918834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.918804 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:51.922292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.922258 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:51.922398 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.922311 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:51.924100 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.923584 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:54:51.925093 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.925067 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "kube-api-access-cqjqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:51.939931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:51.939144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy" pod "054f2f7e-0462-4f88-8cf1-56888d1d20d0" (UID: "054f2f7e-0462-4f88-8cf1-56888d1d20d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:52.024408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024361 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024400 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024413 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024424 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024437 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024448 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054f2f7e-0462-4f88-8cf1-56888d1d20d0-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.024671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.024460 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjqz\" (UniqueName: \"kubernetes.io/projected/054f2f7e-0462-4f88-8cf1-56888d1d20d0-kube-api-access-cqjqz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:52.044037 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.043998 4920 generic.go:356] "Generic (PLEG): container finished" podID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerID="9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279" exitCode=0 Oct 30 09:54:52.044279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.044062 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" Oct 30 09:54:52.044279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.044080 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerDied","Data":"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279"} Oct 30 09:54:52.044279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.044176 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv" event={"ID":"054f2f7e-0462-4f88-8cf1-56888d1d20d0","Type":"ContainerDied","Data":"6283663ab4537aecc7fd8eab25182b0ee26fc0bc2a63229da9a7d7172e1cc82f"} Oct 30 09:54:52.044279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.044199 4920 scope.go:122] "RemoveContainer" containerID="9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279" Oct 30 09:54:52.057043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.057024 4920 scope.go:122] "RemoveContainer" containerID="ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda" Oct 30 09:54:52.068296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.068278 4920 scope.go:122] "RemoveContainer" containerID="9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279" Oct 30 09:54:52.068586 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:54:52.068569 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279\": container with ID starting with 9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279 not found: ID does not exist" containerID="9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279" Oct 30 09:54:52.068641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.068595 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279"} err="failed to get container status \"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279\": rpc error: code = NotFound desc = could not find container \"9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279\": container with ID starting with 9c332d80de1530b02fd67854d21200e7b8e54d09b10d2d4343c095a58da6d279 not found: ID does not exist" Oct 30 09:54:52.068641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.068616 4920 scope.go:122] "RemoveContainer" containerID="ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda" Oct 30 09:54:52.068855 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:54:52.068835 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda\": container with ID starting with ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda not found: ID does not exist" containerID="ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda" Oct 30 09:54:52.068918 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.068866 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda"} err="failed to get container status \"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda\": rpc error: code = NotFound desc = could not find container \"ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda\": container with ID starting with ec5712d1681f8ac3e86274a7387a7fe2dbf81c2d272fc408af6762f833384dda not found: ID does not exist" Oct 30 09:54:52.097505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.097413 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:54:52.103607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.103575 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-035d7d-openshift-default-8957ffc65-vd6wv"] Oct 30 09:54:52.374990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.374908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:52.401255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:52.401201 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" path="/var/lib/kubelet/pods/054f2f7e-0462-4f88-8cf1-56888d1d20d0/volumes" Oct 30 09:54:53.847099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:53.847040 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:54:53.872590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:53.872553 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:54.055505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:54.055468 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:54.195269 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:54.195160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:55.739238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:55.738116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:55.868018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:55.857680 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:56.073287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.073057 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:56.074088 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.074063 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:54:56.082834 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.080732 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" containerID="cri-o://175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23" gracePeriod=30 Oct 30 09:54:56.090121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.089846 4920 generic.go:356] "Generic (PLEG): container finished" podID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerID="2eb7935079e2f2a28bce0f8739dec61d14ed2b792be6d13ad7a82e2b435350e5" exitCode=0 Oct 30 09:54:56.090121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.089995 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerDied","Data":"2eb7935079e2f2a28bce0f8739dec61d14ed2b792be6d13ad7a82e2b435350e5"} Oct 30 09:54:56.090121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.090055 4920 scope.go:122] "RemoveContainer" containerID="a1749ec4924f4e68a929647e1cee753b7e06a3c5750578a9190edb95c9b46eab" Oct 30 09:54:56.668032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.667280 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:54:56.824871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.824828 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.824887 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.824930 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.824962 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825074 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825153 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825196 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825300 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825354 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825374 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.825413 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825359 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:56.826011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825495 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\") pod \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\" (UID: \"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2\") " Oct 30 09:54:56.826011 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.825598 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:56.826168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.826146 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.826270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.826174 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.826270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.826188 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.826270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.826202 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.826270 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.826198 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:56.827925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.827897 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:56.828058 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.827934 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:54:56.828201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.828170 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "kube-api-access-pzs94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:56.828483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.828460 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy" pod "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" (UID: "60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:54:56.928172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.928106 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.928172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.928139 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.928172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.928153 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.928172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.928166 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzs94\" (UniqueName: \"kubernetes.io/projected/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-kube-api-access-pzs94\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:56.928172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:56.928181 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:54:57.099668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.099593 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" Oct 30 09:54:57.099668 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.099633 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n" event={"ID":"60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2","Type":"ContainerDied","Data":"7a1248f2406efc493ddd8e22de0af94127cd967d31ccfb4c4a2bc317775ffad2"} Oct 30 09:54:57.099863 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.099699 4920 scope.go:122] "RemoveContainer" containerID="2eb7935079e2f2a28bce0f8739dec61d14ed2b792be6d13ad7a82e2b435350e5" Oct 30 09:54:57.151876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.151839 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:54:57.163746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.163714 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d60c68-openshift-default-69c4849ff7-8fk6n"] Oct 30 09:54:57.464462 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.464372 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:57.742373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.742282 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:57.839899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:57.839848 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:58.476376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:58.474256 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" path="/var/lib/kubelet/pods/60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2/volumes" Oct 30 09:54:59.113680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:59.113645 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:54:59.225240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:59.224313 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:54:59.225240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:54:59.224400 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:55:00.738860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:00.738814 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:01.142426 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:01.142390 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:55:01.401874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.401848 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:55:01.442456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442419 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442489 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442554 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442602 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442663 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442680 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442740 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442797 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.442853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442820 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\") pod \"61a86afa-224a-40ea-8eba-14c2513182b1\" (UID: \"61a86afa-224a-40ea-8eba-14c2513182b1\") " Oct 30 09:55:01.443135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442857 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:01.443135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.442888 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:01.443135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443065 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:01.443383 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443355 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:01.443724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443691 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:01.443832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443811 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a86afa-224a-40ea-8eba-14c2513182b1-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.443898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443835 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.443898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443853 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.443898 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.443868 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.445717 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.445682 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:01.445835 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.445795 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "kube-api-access-66r8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:01.445969 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.445951 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:55:01.446274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.446258 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy" pod "61a86afa-224a-40ea-8eba-14c2513182b1" (UID: "61a86afa-224a-40ea-8eba-14c2513182b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:01.545932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.545891 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.545932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.545931 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66r8h\" (UniqueName: \"kubernetes.io/projected/61a86afa-224a-40ea-8eba-14c2513182b1-kube-api-access-66r8h\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.546233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.545948 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a86afa-224a-40ea-8eba-14c2513182b1-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.546233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.545961 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.546233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.545975 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a86afa-224a-40ea-8eba-14c2513182b1-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:01.796739 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.796653 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:01.897830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:01.897750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:02.154135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.154045 4920 generic.go:356] "Generic (PLEG): container finished" podID="61a86afa-224a-40ea-8eba-14c2513182b1" containerID="175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23" exitCode=0 Oct 30 09:55:02.154135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.154094 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" Oct 30 09:55:02.154135 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.154118 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerDied","Data":"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23"} Oct 30 09:55:02.154402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.154159 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm" event={"ID":"61a86afa-224a-40ea-8eba-14c2513182b1","Type":"ContainerDied","Data":"c10618b69205ae5aeb7976ddfff11f36d7db514e6c334ff32f16c1f5c457ec76"} Oct 30 09:55:02.154402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.154181 4920 scope.go:122] "RemoveContainer" containerID="175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23" Oct 30 09:55:02.166672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.166651 4920 scope.go:122] "RemoveContainer" containerID="7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c" Oct 30 09:55:02.177302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.177281 4920 scope.go:122] "RemoveContainer" containerID="175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23" Oct 30 09:55:02.177602 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:02.177578 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23\": container with ID starting with 175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23 not found: ID does not exist" containerID="175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23" Oct 30 09:55:02.177655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.177612 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23"} err="failed to get container status \"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23\": rpc error: code = NotFound desc = could not find container \"175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23\": container with ID starting with 175d4e353655c854f528cd3a439939aac8b8034794267d61b5383ad09a545c23 not found: ID does not exist" Oct 30 09:55:02.177655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.177631 4920 scope.go:122] "RemoveContainer" containerID="7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c" Oct 30 09:55:02.177892 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:02.177870 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c\": container with ID starting with 7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c not found: ID does not exist" containerID="7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c" Oct 30 09:55:02.177936 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.177900 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c"} err="failed to get container status \"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c\": rpc error: code = NotFound desc = could not find container \"7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c\": container with ID starting with 7b0490f9016fd7fe85647157d4c1b02a17e850e1a14454bc4bbe643a8b94a15c not found: ID does not exist" Oct 30 09:55:02.218200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.218157 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:55:02.230188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.230150 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-88cbc7-openshift-default-74778845d9-6lctm"] Oct 30 09:55:02.253941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.253896 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:02.448286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:02.446998 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" path="/var/lib/kubelet/pods/61a86afa-224a-40ea-8eba-14c2513182b1/volumes" Oct 30 09:55:03.769748 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:03.769649 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:05.308689 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:05.308644 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:06.856529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:06.856497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:07.423311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:07.421582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:07.549773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:07.549736 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:08.569027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:08.568991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:10.151605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:10.151568 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:11.640132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:11.640097 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:13.333789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:13.333753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:14.849308 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:14.849199 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:16.486329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:16.486276 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:17.976889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:17.976840 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:19.209180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.209140 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:55:19.209709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.209407 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" podUID="ebf964b5-9081-43a1-8345-f60fb1a375eb" containerName="maas-api" containerID="cri-o://ca770ee7a68847850a7cfeba8abfed67eb5705ae4912ef4de6c9dc91e9673e82" gracePeriod=30 Oct 30 09:55:19.253905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.253867 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:19.366875 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.366838 4920 generic.go:356] "Generic (PLEG): container finished" podID="ebf964b5-9081-43a1-8345-f60fb1a375eb" containerID="ca770ee7a68847850a7cfeba8abfed67eb5705ae4912ef4de6c9dc91e9673e82" exitCode=0 Oct 30 09:55:19.367096 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.366922 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" event={"ID":"ebf964b5-9081-43a1-8345-f60fb1a375eb","Type":"ContainerDied","Data":"ca770ee7a68847850a7cfeba8abfed67eb5705ae4912ef4de6c9dc91e9673e82"} Oct 30 09:55:19.902920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.894818 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:19.920780 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.917479 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:55:19.964033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.963999 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\") pod \"ebf964b5-9081-43a1-8345-f60fb1a375eb\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " Oct 30 09:55:19.964203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.964056 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\") pod \"ebf964b5-9081-43a1-8345-f60fb1a375eb\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " Oct 30 09:55:19.964203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.964098 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\" (UniqueName: \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\") pod \"ebf964b5-9081-43a1-8345-f60fb1a375eb\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " Oct 30 09:55:19.964203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.964134 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\") pod \"ebf964b5-9081-43a1-8345-f60fb1a375eb\" (UID: \"ebf964b5-9081-43a1-8345-f60fb1a375eb\") " Oct 30 09:55:19.964606 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.964572 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca" pod "ebf964b5-9081-43a1-8345-f60fb1a375eb" (UID: "ebf964b5-9081-43a1-8345-f60fb1a375eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:19.964786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.964767 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf964b5-9081-43a1-8345-f60fb1a375eb-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:19.966824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.966792 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs" pod "ebf964b5-9081-43a1-8345-f60fb1a375eb" (UID: "ebf964b5-9081-43a1-8345-f60fb1a375eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:19.966939 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.966887 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd" pod "ebf964b5-9081-43a1-8345-f60fb1a375eb" (UID: "ebf964b5-9081-43a1-8345-f60fb1a375eb"). InnerVolumeSpecName "kube-api-access-jtnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:19.967018 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.966999 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls" pod "ebf964b5-9081-43a1-8345-f60fb1a375eb" (UID: "ebf964b5-9081-43a1-8345-f60fb1a375eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:19.977326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:19.977285 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:20.066402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.066367 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:20.066402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.066400 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnvd\" (UniqueName: \"kubernetes.io/projected/ebf964b5-9081-43a1-8345-f60fb1a375eb-kube-api-access-jtnvd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:20.066402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.066412 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf964b5-9081-43a1-8345-f60fb1a375eb-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:20.377367 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.377337 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" Oct 30 09:55:20.377792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.377336 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz" event={"ID":"ebf964b5-9081-43a1-8345-f60fb1a375eb","Type":"ContainerDied","Data":"4eb07ed2275e108276d86da08ec8503b9738ee83f7b6377389c6e32a4fbf0d6b"} Oct 30 09:55:20.377792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.377476 4920 scope.go:122] "RemoveContainer" containerID="ca770ee7a68847850a7cfeba8abfed67eb5705ae4912ef4de6c9dc91e9673e82" Oct 30 09:55:20.428734 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.428681 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:20.432024 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.431991 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:55:20.443709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:20.443679 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fc7ee7-b6b7fcff5-sjdgz"] Oct 30 09:55:21.560553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:21.560497 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:22.401531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:22.401491 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf964b5-9081-43a1-8345-f60fb1a375eb" path="/var/lib/kubelet/pods/ebf964b5-9081-43a1-8345-f60fb1a375eb/volumes" Oct 30 09:55:23.160284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:23.160246 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:24.618364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:24.618325 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:26.280662 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:26.280622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:26.697062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:26.697015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:26.802008 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:26.801968 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:27.035280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:27.035250 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:28.086311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:28.086276 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:29.695592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:29.695554 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:31.139750 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:31.139700 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:55:31.345000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:31.344964 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:32.634932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:32.634898 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:33.200204 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:33.200165 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:33.332263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:33.332176 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:33.436172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:33.436123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:33.893757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:33.893722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:35.101068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:35.101022 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:35.943476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:35.943433 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:55:36.463264 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:36.459554 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:37.216167 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:37.216122 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:37.333783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:37.333748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:37.685065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:37.685016 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:38.344419 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:38.344374 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:38.477549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:38.477507 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:39.225177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:39.225135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:40.138831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:40.138795 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:40.227818 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:40.227777 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:40.477528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:40.477435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:40.795014 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:40.794980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:42.464233 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:42.464184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:43.601478 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:43.601442 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:43.716656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:43.716619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:43.945805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:43.945714 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:55:43.946063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:43.946031 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" containerID="cri-o://ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7" gracePeriod=30 Oct 30 09:55:43.952503 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:43.952475 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:44.061815 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:44.061766 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:45.666824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:45.666788 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:47.406675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:47.406626 4920 patch_prober.go:29] interesting pod/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:55:47.407187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:47.406708 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:55:47.440569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:47.440523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:49.111532 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.111493 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:49.311456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.311423 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:55:49.471134 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471103 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471204 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471263 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471317 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471341 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471362 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471441 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.471487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471459 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471554 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.471707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471583 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471623 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.471707 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471694 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\") pod \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\" (UID: \"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f\") " Oct 30 09:55:49.471929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471713 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.471929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.471850 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:49.472557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.472534 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.472557 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.472559 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.472765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.472574 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.472765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.472586 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.472765 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.472597 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.474252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.474087 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "kube-api-access-p7wft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:49.474252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.474137 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:55:49.474252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.474157 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:55:49.474468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.474450 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token" pod "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" (UID: "a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:49.574129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.574093 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.574129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.574125 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.574129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.574136 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.574432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.574146 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wft\" (UniqueName: \"kubernetes.io/projected/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f-kube-api-access-p7wft\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:55:49.810195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.810158 4920 generic.go:356] "Generic (PLEG): container finished" podID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerID="ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7" exitCode=0 Oct 30 09:55:49.810449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.810266 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" Oct 30 09:55:49.810449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.810306 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerDied","Data":"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7"} Oct 30 09:55:49.810449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.810362 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28" event={"ID":"a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f","Type":"ContainerDied","Data":"faa78504a7a305c60175db3756bd19aae3a2dd5cc512f9ba0a1943349546ece2"} Oct 30 09:55:49.810449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.810385 4920 scope.go:122] "RemoveContainer" containerID="ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7" Oct 30 09:55:49.823173 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.823150 4920 scope.go:122] "RemoveContainer" containerID="6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083" Oct 30 09:55:49.835736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.835695 4920 scope.go:122] "RemoveContainer" containerID="ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7" Oct 30 09:55:49.835946 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:49.835923 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7\": container with ID starting with ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7 not found: ID does not exist" containerID="ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7" Oct 30 09:55:49.836054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.835958 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7"} err="failed to get container status \"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7\": rpc error: code = NotFound desc = could not find container \"ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7\": container with ID starting with ad03015765fe1a6bf5c2fd1ac859fa4bbe09d265dc46a53192622814a02437a7 not found: ID does not exist" Oct 30 09:55:49.836054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.835982 4920 scope.go:122] "RemoveContainer" containerID="6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083" Oct 30 09:55:49.836341 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:55:49.836304 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083\": container with ID starting with 6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083 not found: ID does not exist" containerID="6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083" Oct 30 09:55:49.836420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.836350 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083"} err="failed to get container status \"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083\": rpc error: code = NotFound desc = could not find container \"6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083\": container with ID starting with 6e340c8e68c4f0c003d7baee968e74db25d3973f3682375eb2e4e29bb9e85083 not found: ID does not exist" Oct 30 09:55:49.865249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.864249 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:55:49.880303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:49.880265 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d66d22-openshift-default-64984c7669-rjq28"] Oct 30 09:55:50.399422 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:50.399382 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" path="/var/lib/kubelet/pods/a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f/volumes" Oct 30 09:55:50.829677 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:50.829550 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:51.641704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:51.641663 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:51.765742 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:51.765700 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:52.462411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:52.462376 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:54.069126 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:54.069088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:54.183925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:54.183890 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:55.144314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:55.144258 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:55.268071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:55.268031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:55.580428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:55.580397 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:57.051992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:57.051957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:55:58.594706 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:55:58.594663 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:00.100059 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:00.100026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:01.137116 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:56:01.137072 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:56:01.655481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:01.655450 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:03.104258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:03.104184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:04.579395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:04.579352 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:06.179920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:06.179880 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:07.669841 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:07.669807 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:09.238070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:09.238035 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:10.502822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:10.502789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:10.909243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:10.904621 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:11.093845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:11.093759 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:11.192744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:11.192594 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:11.643437 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:11.643400 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:11.744840 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:11.744797 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:11.829687 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:11.829653 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:12.250620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:12.250589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:13.496757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:13.496719 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:56:13.522347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:13.522312 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:15.243517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:15.243477 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:16.769240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:16.764439 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:18.318581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.318543 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:18.426322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.426281 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:18.535106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.535069 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:18.686160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.686066 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:18.698239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.694899 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:56:18.698239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.695133 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" containerID="cri-o://5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0" gracePeriod=30 Oct 30 09:56:18.855240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:18.831722 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:19.152331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:19.152293 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:20.295528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:20.295491 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:21.709118 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:21.709074 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:56:21.709764 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:21.709130 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:56:21.777268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:21.777229 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:23.355041 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:23.354998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:24.013445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.013408 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:56:24.129302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.129266 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:56:24.138563 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138525 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.138757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138698 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.138757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138729 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.138883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138808 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.138883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138818 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139019 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.138887 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139026 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139172 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139189 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:24.139277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139253 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139393 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139297 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\") pod \"32beebe9-cdc6-464b-af7e-50bce18f65f9\" (UID: \"32beebe9-cdc6-464b-af7e-50bce18f65f9\") " Oct 30 09:56:24.139465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139441 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.139655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139627 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.139947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.139925 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.140510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.140480 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.140605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.140515 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.140605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.140532 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.140605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.140565 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32beebe9-cdc6-464b-af7e-50bce18f65f9-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.140605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.140580 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.141829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.141796 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.142060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.142035 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:24.142060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.142052 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "kube-api-access-kpjdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.142272 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.142126 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token" pod "32beebe9-cdc6-464b-af7e-50bce18f65f9" (UID: "32beebe9-cdc6-464b-af7e-50bce18f65f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.152211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.152180 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:56:24.234915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.234828 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:24.242553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.242522 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.242553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.242552 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.242711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.242563 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjdp\" (UniqueName: \"kubernetes.io/projected/32beebe9-cdc6-464b-af7e-50bce18f65f9-kube-api-access-kpjdp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.242711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.242574 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32beebe9-cdc6-464b-af7e-50bce18f65f9-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.350320 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.350285 4920 generic.go:356] "Generic (PLEG): container finished" podID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerID="5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0" exitCode=0 Oct 30 09:56:24.350510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.350344 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" Oct 30 09:56:24.350510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.350351 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" event={"ID":"32beebe9-cdc6-464b-af7e-50bce18f65f9","Type":"ContainerDied","Data":"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0"} Oct 30 09:56:24.350510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.350406 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68" event={"ID":"32beebe9-cdc6-464b-af7e-50bce18f65f9","Type":"ContainerDied","Data":"c52f37a543f894ecd5d4ee9eac14dfcdcf5d280c84eca22348534f9d38ba05de"} Oct 30 09:56:24.350510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.350437 4920 scope.go:122] "RemoveContainer" containerID="5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0" Oct 30 09:56:24.363337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.363316 4920 scope.go:122] "RemoveContainer" containerID="5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0" Oct 30 09:56:24.363653 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:56:24.363619 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0\": container with ID starting with 5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0 not found: ID does not exist" containerID="5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0" Oct 30 09:56:24.363698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.363655 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0"} err="failed to get container status \"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0\": rpc error: code = NotFound desc = could not find container \"5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0\": container with ID starting with 5a5e47dd9b8dd42d08ff6a95ad803086e451625ac3a16fcf0c3a5cb8888bc9d0 not found: ID does not exist" Oct 30 09:56:24.406029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.405990 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:56:24.413989 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.413955 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-5wb68"] Oct 30 09:56:24.735594 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:24.735558 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:25.316166 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:25.316112 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:25.455991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:25.455946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:25.547854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:25.547816 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:25.972553 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:25.972519 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:26.399388 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:26.399353 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" path="/var/lib/kubelet/pods/32beebe9-cdc6-464b-af7e-50bce18f65f9/volumes" Oct 30 09:56:27.242238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:27.240504 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:28.747322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:28.747278 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:29.695442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:29.695405 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:56:29.792254 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:29.791608 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:29.891917 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:29.891872 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:30.249950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:30.249916 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:30.610148 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:30.610104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:30.702180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:30.702143 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:31.348506 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:56:31.348439 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:56:31.808588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:31.808553 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:31.909191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:31.909155 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:32.018323 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:32.015679 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:32.164905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:32.164821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:32.168642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:32.168600 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:56:32.169174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:32.169131 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" containerID="cri-o://0a697d3e0334a005fc15a1ee082ab115e805fc69f1ec93e3a7a525b9e256c71c" gracePeriod=30 Oct 30 09:56:33.601374 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:33.601338 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:34.696108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:34.696067 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" containerID="cri-o://3c676eb1e286c4c9005cfa316c1af72e6d80091fadf0af06f6cbabfd05c0ff49" gracePeriod=25 Oct 30 09:56:35.279592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:35.279551 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:36.760540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:36.760506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:37.614124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:37.614086 4920 generic.go:356] "Generic (PLEG): container finished" podID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerID="0a697d3e0334a005fc15a1ee082ab115e805fc69f1ec93e3a7a525b9e256c71c" exitCode=0 Oct 30 09:56:37.614452 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:37.614414 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" event={"ID":"5b7795d4-23df-4ff0-8d58-89858fec7b26","Type":"ContainerDied","Data":"0a697d3e0334a005fc15a1ee082ab115e805fc69f1ec93e3a7a525b9e256c71c"} Oct 30 09:56:37.894506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:37.894429 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:56:38.000342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000298 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.000536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000428 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.000590 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000554 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.000642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000597 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.000698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000653 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:38.000819 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000794 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:38.000966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000943 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:38.001026 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.000764 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.001078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.001049 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.001176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.001159 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.001274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.001205 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.001363 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.001292 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\") pod \"5b7795d4-23df-4ff0-8d58-89858fec7b26\" (UID: \"5b7795d4-23df-4ff0-8d58-89858fec7b26\") " Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.001649 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002005 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002210 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002251 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7795d4-23df-4ff0-8d58-89858fec7b26-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002266 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002281 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.002321 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.002296 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.004262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.003843 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:38.004262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.003861 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "kube-api-access-h9v8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:38.004262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.003893 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:38.005492 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.005465 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy" pod "5b7795d4-23df-4ff0-8d58-89858fec7b26" (UID: "5b7795d4-23df-4ff0-8d58-89858fec7b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:38.103654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.103620 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.103654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.103651 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9v8p\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-kube-api-access-h9v8p\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.103654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.103661 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.103905 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.103670 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7795d4-23df-4ff0-8d58-89858fec7b26-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.454568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.454533 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:38.657268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.656650 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" event={"ID":"5b7795d4-23df-4ff0-8d58-89858fec7b26","Type":"ContainerDied","Data":"adeeba2ef774d1ff511834f4ee02552dc0bd14607c2793638d286d89490bb002"} Oct 30 09:56:38.657268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.656746 4920 scope.go:122] "RemoveContainer" containerID="0a697d3e0334a005fc15a1ee082ab115e805fc69f1ec93e3a7a525b9e256c71c" Oct 30 09:56:38.657268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.657062 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z" Oct 30 09:56:38.743603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.743563 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:56:38.743990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:38.743968 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-znr6z"] Oct 30 09:56:39.155202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.155157 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" containerID="cri-o://d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb" gracePeriod=45 Oct 30 09:56:39.174795 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.174757 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" containerID="cri-o://5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de" gracePeriod=45 Oct 30 09:56:39.514396 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.514362 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:56:39.549851 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.549782 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:56:39.561977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.561938 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.562177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.562057 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.562177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.562093 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.562177 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.562161 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.562542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.562206 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.562542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.562293 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\" (UniqueName: \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\") pod \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\" (UID: \"4878df82-03f8-4d95-8173-9cb84d7a8c8c\") " Oct 30 09:56:39.563620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.563539 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.563620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.563551 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.563796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.563624 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.566030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.566002 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "kube-api-access-d5wmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:39.566470 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.566420 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.566604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.566585 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs" pod "4878df82-03f8-4d95-8173-9cb84d7a8c8c" (UID: "4878df82-03f8-4d95-8173-9cb84d7a8c8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:39.663738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.663688 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.663945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.663781 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\" (UniqueName: \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.663945 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.663874 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.664083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.663959 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\" (UniqueName: \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.664083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664002 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.664083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664031 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\") pod \"297160b5-6012-4f00-81c8-c01911769694\" (UID: \"297160b5-6012-4f00-81c8-c01911769694\") " Oct 30 09:56:39.664284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664072 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664432 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664449 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664458 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878df82-03f8-4d95-8173-9cb84d7a8c8c-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664467 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wmd\" (UniqueName: \"kubernetes.io/projected/4878df82-03f8-4d95-8173-9cb84d7a8c8c-kube-api-access-d5wmd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664475 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664483 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664490 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878df82-03f8-4d95-8173-9cb84d7a8c8c-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664755 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.664959 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.664885 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.666874 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.666847 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.666995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.666913 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "kube-api-access-z6kxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:39.667276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.667212 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs" pod "297160b5-6012-4f00-81c8-c01911769694" (UID: "297160b5-6012-4f00-81c8-c01911769694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:39.670578 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.670551 4920 generic.go:356] "Generic (PLEG): container finished" podID="297160b5-6012-4f00-81c8-c01911769694" containerID="5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de" exitCode=0 Oct 30 09:56:39.670711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.670610 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" Oct 30 09:56:39.670711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.670634 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" event={"ID":"297160b5-6012-4f00-81c8-c01911769694","Type":"ContainerDied","Data":"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de"} Oct 30 09:56:39.670711 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.670689 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" event={"ID":"297160b5-6012-4f00-81c8-c01911769694","Type":"ContainerDied","Data":"164cad662e9e33702079120a5140bdbf71bccccfa9bc26c4591b824df056b312"} Oct 30 09:56:39.670887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.670713 4920 scope.go:122] "RemoveContainer" containerID="5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de" Oct 30 09:56:39.672300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.672273 4920 generic.go:356] "Generic (PLEG): container finished" podID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerID="d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb" exitCode=0 Oct 30 09:56:39.672407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.672316 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" Oct 30 09:56:39.672407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.672352 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" event={"ID":"4878df82-03f8-4d95-8173-9cb84d7a8c8c","Type":"ContainerDied","Data":"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb"} Oct 30 09:56:39.672407 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.672381 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n" event={"ID":"4878df82-03f8-4d95-8173-9cb84d7a8c8c","Type":"ContainerDied","Data":"80fdd7b76181a76c6e3200a2d55845543b6a459db1e109dbc80888829687195e"} Oct 30 09:56:39.683175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.683150 4920 scope.go:122] "RemoveContainer" containerID="5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de" Oct 30 09:56:39.683639 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:56:39.683546 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de\": container with ID starting with 5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de not found: ID does not exist" containerID="5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de" Oct 30 09:56:39.683639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.683579 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de"} err="failed to get container status \"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de\": rpc error: code = NotFound desc = could not find container \"5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de\": container with ID starting with 5527e2c7fd725ca2cdbbe515e24b42303ac247ef3c474a7b4b509be341f4a0de not found: ID does not exist" Oct 30 09:56:39.683639 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.683603 4920 scope.go:122] "RemoveContainer" containerID="d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb" Oct 30 09:56:39.695365 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.695340 4920 scope.go:122] "RemoveContainer" containerID="d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb" Oct 30 09:56:39.695688 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:56:39.695659 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb\": container with ID starting with d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb not found: ID does not exist" containerID="d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb" Oct 30 09:56:39.695824 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.695691 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb"} err="failed to get container status \"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb\": rpc error: code = NotFound desc = could not find container \"d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb\": container with ID starting with d6ce6a79851d83f10a887a15b527f57ada050450ea58f736ed2017915c65dbbb not found: ID does not exist" Oct 30 09:56:39.721708 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.721624 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:56:39.726826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.726798 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-w1-c5447d-kserve-6d5dd79bb7-69n7n"] Oct 30 09:56:39.765582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.765544 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297160b5-6012-4f00-81c8-c01911769694-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.765582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.765576 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.765582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.765588 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6kxn\" (UniqueName: \"kubernetes.io/projected/297160b5-6012-4f00-81c8-c01911769694-kube-api-access-z6kxn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.765899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.765599 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.765899 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.765610 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297160b5-6012-4f00-81c8-c01911769694-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.772158 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.772128 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:56:39.786889 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:39.786851 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25"] Oct 30 09:56:40.065282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:40.065246 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:40.402317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:40.402200 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297160b5-6012-4f00-81c8-c01911769694" path="/var/lib/kubelet/pods/297160b5-6012-4f00-81c8-c01911769694/volumes" Oct 30 09:56:40.403020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:40.402985 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" path="/var/lib/kubelet/pods/4878df82-03f8-4d95-8173-9cb84d7a8c8c/volumes" Oct 30 09:56:40.403723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:40.403701 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" path="/var/lib/kubelet/pods/5b7795d4-23df-4ff0-8d58-89858fec7b26/volumes" Oct 30 09:56:40.457397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:40.457359 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/simulator-w1-c5447d-kserve-7f9c5df5d9-v2r25" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:56:41.242340 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:41.242292 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:41.597292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:41.595896 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:43.167588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:43.167551 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:43.332371 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:43.332334 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:43.481291 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:43.469174 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:44.764438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:44.764402 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:46.257991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:46.257956 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:47.564282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.564244 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:56:47.564794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.564484 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerName="maas-api" containerID="cri-o://32a84569db7e9881c9b92dd6b2146841d8ac51a323d5bf94599cf0cf876fa1cc" gracePeriod=30 Oct 30 09:56:47.613386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.613351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:47.727347 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.727305 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.225:8443/health\": dial tcp 10.132.0.225:8443: connect: connection refused" Oct 30 09:56:47.774638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.774604 4920 generic.go:356] "Generic (PLEG): container finished" podID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerID="32a84569db7e9881c9b92dd6b2146841d8ac51a323d5bf94599cf0cf876fa1cc" exitCode=0 Oct 30 09:56:47.774838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:47.774704 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" event={"ID":"7ca2f98d-2b90-4d40-82bf-abb15f0f7105","Type":"ContainerDied","Data":"32a84569db7e9881c9b92dd6b2146841d8ac51a323d5bf94599cf0cf876fa1cc"} Oct 30 09:56:48.135919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.134611 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:48.152700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.144545 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:56:48.204261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.204176 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\") pod \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " Oct 30 09:56:48.204476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.204296 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\") pod \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " Oct 30 09:56:48.204476 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.204353 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\") pod \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " Oct 30 09:56:48.204610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.204519 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\" (UniqueName: \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\") pod \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\" (UID: \"7ca2f98d-2b90-4d40-82bf-abb15f0f7105\") " Oct 30 09:56:48.205062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.205033 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca" pod "7ca2f98d-2b90-4d40-82bf-abb15f0f7105" (UID: "7ca2f98d-2b90-4d40-82bf-abb15f0f7105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:48.205442 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.205409 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:48.211052 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.211023 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls" pod "7ca2f98d-2b90-4d40-82bf-abb15f0f7105" (UID: "7ca2f98d-2b90-4d40-82bf-abb15f0f7105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:48.211174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.211111 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb" pod "7ca2f98d-2b90-4d40-82bf-abb15f0f7105" (UID: "7ca2f98d-2b90-4d40-82bf-abb15f0f7105"). InnerVolumeSpecName "kube-api-access-sqswb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:48.212904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.212828 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs" pod "7ca2f98d-2b90-4d40-82bf-abb15f0f7105" (UID: "7ca2f98d-2b90-4d40-82bf-abb15f0f7105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:48.230339 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.230306 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:48.307182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.307142 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqswb\" (UniqueName: \"kubernetes.io/projected/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-kube-api-access-sqswb\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:48.307182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.307182 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:48.307469 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.307198 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca2f98d-2b90-4d40-82bf-abb15f0f7105-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:48.328823 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.328771 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:48.688055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.688025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:48.783715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.783675 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" event={"ID":"7ca2f98d-2b90-4d40-82bf-abb15f0f7105","Type":"ContainerDied","Data":"c21aeb2e88b7a4b8e9ea14f8bfb3c1c0c85707239a677260013c83ac70f30186"} Oct 30 09:56:48.783920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.783743 4920 scope.go:122] "RemoveContainer" containerID="32a84569db7e9881c9b92dd6b2146841d8ac51a323d5bf94599cf0cf876fa1cc" Oct 30 09:56:48.783920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.783692 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w" Oct 30 09:56:48.838854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.838769 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:56:48.848983 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:48.848953 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0dc148-c66787c98-svw5w"] Oct 30 09:56:49.096294 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:49.096189 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:56:49.124336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:49.124298 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:50.044200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:50.044163 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:50.400712 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:50.400625 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" path="/var/lib/kubelet/pods/7ca2f98d-2b90-4d40-82bf-abb15f0f7105/volumes" Oct 30 09:56:51.243385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:51.243340 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:51.557669 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:51.557604 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:53.258035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:53.257996 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:54.256604 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.256547 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:54.763260 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.763202 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:54.867246 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.867152 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:54.887207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.887165 4920 generic.go:356] "Generic (PLEG): container finished" podID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerID="3c676eb1e286c4c9005cfa316c1af72e6d80091fadf0af06f6cbabfd05c0ff49" exitCode=0 Oct 30 09:56:54.887497 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.887344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" event={"ID":"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37","Type":"ContainerDied","Data":"3c676eb1e286c4c9005cfa316c1af72e6d80091fadf0af06f6cbabfd05c0ff49"} Oct 30 09:56:54.966794 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:54.966710 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:55.133582 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.133548 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:56:55.160086 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.160051 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:56:55.160351 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.160327 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" containerID="cri-o://ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec" gracePeriod=30 Oct 30 09:56:55.183397 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.183362 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:55.275300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.270689 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\") pod \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " Oct 30 09:56:55.275300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.270878 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\" (UniqueName: \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\") pod \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\" (UID: \"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37\") " Oct 30 09:56:55.311244 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.287313 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume" pod "7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" (UID: "7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.357307 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.349211 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8" pod "7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" (UID: "7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37"). InnerVolumeSpecName "kube-api-access-8kjm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.375757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.375659 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.375757 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.375695 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjm8\" (UniqueName: \"kubernetes.io/projected/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37-kube-api-access-8kjm8\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.647758 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.647715 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:55.901796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.901700 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" event={"ID":"7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37","Type":"ContainerDied","Data":"88f9c87c4ea1faf66264f184bf10ef2921a9947d996d79ecd8e7272430a4fce2"} Oct 30 09:56:55.901796 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.901778 4920 scope.go:122] "RemoveContainer" containerID="3c676eb1e286c4c9005cfa316c1af72e6d80091fadf0af06f6cbabfd05c0ff49" Oct 30 09:56:55.902377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.901942 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6" Oct 30 09:56:55.956520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.956483 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:56:55.965513 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:55.965482 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c5447d-b8dd68d55-n6rd6"] Oct 30 09:56:56.402068 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:56.402033 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" path="/var/lib/kubelet/pods/7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37/volumes" Oct 30 09:56:56.520199 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:56.520160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:57.706852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:57.706814 4920 patch_prober.go:29] interesting pod/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:56:57.707286 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:57.706883 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:56:58.272679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:58.272643 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:56:59.826942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:56:59.826896 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:00.481910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.481881 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:57:00.652771 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.652731 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.652970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.652819 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.652970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.652945 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.652993 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653077 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653023 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653075 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653128 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:00.653202 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653179 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:00.653520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653199 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653300 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653353 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\") pod \"36bd5027-e611-438c-a0cb-c3b092154d30\" (UID: \"36bd5027-e611-438c-a0cb-c3b092154d30\") " Oct 30 09:57:00.653520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653352 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:00.653520 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653438 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:00.653791 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653691 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:00.653991 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653972 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.654054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.653996 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.654054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.654010 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.654054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.654019 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.654054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.654027 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bd5027-e611-438c-a0cb-c3b092154d30-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.655750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.655729 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:00.655859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.655791 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "kube-api-access-97m9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:00.655859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.655841 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:00.655992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.655865 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token" pod "36bd5027-e611-438c-a0cb-c3b092154d30" (UID: "36bd5027-e611-438c-a0cb-c3b092154d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:00.746903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.746810 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:57:00.755438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.755408 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.755438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.755436 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bd5027-e611-438c-a0cb-c3b092154d30-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.755438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.755445 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bd5027-e611-438c-a0cb-c3b092154d30-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.755697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.755456 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97m9s\" (UniqueName: \"kubernetes.io/projected/36bd5027-e611-438c-a0cb-c3b092154d30-kube-api-access-97m9s\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:00.815613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.815578 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:00.954770 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.954735 4920 generic.go:356] "Generic (PLEG): container finished" podID="36bd5027-e611-438c-a0cb-c3b092154d30" containerID="ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec" exitCode=0 Oct 30 09:57:00.955193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.954785 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" Oct 30 09:57:00.955193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.954813 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerDied","Data":"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec"} Oct 30 09:57:00.955193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.954895 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb" event={"ID":"36bd5027-e611-438c-a0cb-c3b092154d30","Type":"ContainerDied","Data":"1db02d3a2a651c853035a874e4aee5a6249d87bea83491a363a1c7ef7bc3a6e7"} Oct 30 09:57:00.955193 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.954921 4920 scope.go:122] "RemoveContainer" containerID="ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec" Oct 30 09:57:00.966549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.966530 4920 scope.go:122] "RemoveContainer" containerID="9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8" Oct 30 09:57:00.979027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.978980 4920 scope.go:122] "RemoveContainer" containerID="ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec" Oct 30 09:57:00.979456 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:00.979423 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec\": container with ID starting with ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec not found: ID does not exist" containerID="ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec" Oct 30 09:57:00.979546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.979466 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec"} err="failed to get container status \"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec\": rpc error: code = NotFound desc = could not find container \"ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec\": container with ID starting with ebcf1047cc4c5b62b504cd52a6e3ae20fc5d9603e01b2f1efe1de2e28dac31ec not found: ID does not exist" Oct 30 09:57:00.979546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.979490 4920 scope.go:122] "RemoveContainer" containerID="9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8" Oct 30 09:57:00.979793 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:00.979773 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8\": container with ID starting with 9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8 not found: ID does not exist" containerID="9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8" Oct 30 09:57:00.979853 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:00.979803 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8"} err="failed to get container status \"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8\": rpc error: code = NotFound desc = could not find container \"9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8\": container with ID starting with 9c589aff8a94b3e0acfdf0672a5dddfda65859acc4928c4787b6c949d1610ed8 not found: ID does not exist" Oct 30 09:57:01.006257 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.006129 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:57:01.010301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.010268 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0dc148-openshift-default-5868c4797-sqsxb"] Oct 30 09:57:01.139973 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:01.139939 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:57:01.414986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.414945 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:01.517325 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.516631 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:01.610282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.610207 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:01.918015 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:01.917980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:02.400187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:02.400151 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" path="/var/lib/kubelet/pods/36bd5027-e611-438c-a0cb-c3b092154d30/volumes" Oct 30 09:57:02.868682 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:02.868649 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:57:02.868891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:02.868869 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" podUID="1b6595eb-f73c-4c26-89c9-3a693410528f" containerName="maas-api" containerID="cri-o://c787e7d5460c53ceb939139fb4089c67b3ece26aaede7effa87793efd8e03107" gracePeriod=30 Oct 30 09:57:02.931703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:02.929634 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:03.485783 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.478361 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:03.537314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.537110 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:57:03.584087 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.584051 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:03.655416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.655378 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\") pod \"1b6595eb-f73c-4c26-89c9-3a693410528f\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " Oct 30 09:57:03.655617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.655475 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\" (UniqueName: \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\") pod \"1b6595eb-f73c-4c26-89c9-3a693410528f\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " Oct 30 09:57:03.655690 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.655640 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\") pod \"1b6595eb-f73c-4c26-89c9-3a693410528f\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " Oct 30 09:57:03.655754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.655713 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\") pod \"1b6595eb-f73c-4c26-89c9-3a693410528f\" (UID: \"1b6595eb-f73c-4c26-89c9-3a693410528f\") " Oct 30 09:57:03.656060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.656034 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca" pod "1b6595eb-f73c-4c26-89c9-3a693410528f" (UID: "1b6595eb-f73c-4c26-89c9-3a693410528f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:03.656342 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.656321 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6595eb-f73c-4c26-89c9-3a693410528f-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:03.657637 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.657611 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs" pod "1b6595eb-f73c-4c26-89c9-3a693410528f" (UID: "1b6595eb-f73c-4c26-89c9-3a693410528f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:03.657726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.657666 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls" pod "1b6595eb-f73c-4c26-89c9-3a693410528f" (UID: "1b6595eb-f73c-4c26-89c9-3a693410528f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:03.657726 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.657687 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz" pod "1b6595eb-f73c-4c26-89c9-3a693410528f" (UID: "1b6595eb-f73c-4c26-89c9-3a693410528f"). InnerVolumeSpecName "kube-api-access-pl9sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:03.676699 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.676671 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:03.757680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.757562 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:03.757680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.757594 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9sz\" (UniqueName: \"kubernetes.io/projected/1b6595eb-f73c-4c26-89c9-3a693410528f-kube-api-access-pl9sz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:03.757680 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.757605 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6595eb-f73c-4c26-89c9-3a693410528f-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:03.846411 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.846376 4920 scope.go:122] "RemoveContainer" containerID="c787e7d5460c53ceb939139fb4089c67b3ece26aaede7effa87793efd8e03107" Oct 30 09:57:03.986856 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.986823 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" event={"ID":"1b6595eb-f73c-4c26-89c9-3a693410528f","Type":"ContainerDied","Data":"c787e7d5460c53ceb939139fb4089c67b3ece26aaede7effa87793efd8e03107"} Oct 30 09:57:03.987043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.986875 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" event={"ID":"1b6595eb-f73c-4c26-89c9-3a693410528f","Type":"ContainerDied","Data":"8f51dc1cf7e1822d126645d4bceabe1c3759fa17603d9b429abd826ed34ccd19"} Oct 30 09:57:03.987043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:03.986927 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582" Oct 30 09:57:04.060982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.060950 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:04.063313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.063291 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:57:04.066960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.066933 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7a213d-6b7799b4d5-58582"] Oct 30 09:57:04.120176 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.120108 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerName="main" containerID="cri-o://84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee" gracePeriod=45 Oct 30 09:57:04.400658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.400140 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6595eb-f73c-4c26-89c9-3a693410528f" path="/var/lib/kubelet/pods/1b6595eb-f73c-4c26-89c9-3a693410528f/volumes" Oct 30 09:57:04.403529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.403505 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:57:04.467424 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467385 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\" (UniqueName: \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.467698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467570 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.467698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467630 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.467698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467680 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.467894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467734 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.467894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.467811 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\") pod \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\" (UID: \"61b96941-be95-4b6c-bd41-8d8cef2f4fd6\") " Oct 30 09:57:04.468312 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.468286 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:04.468727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.468701 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:04.468833 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.468770 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:04.468994 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.468973 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.470729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.470694 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:04.470968 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.470945 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:04.471678 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.471657 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht" pod "61b96941-be95-4b6c-bd41-8d8cef2f4fd6" (UID: "61b96941-be95-4b6c-bd41-8d8cef2f4fd6"). InnerVolumeSpecName "kube-api-access-2slht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:04.570623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.570586 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.570623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.570616 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.570623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.570625 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.571097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.570636 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.571097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.570645 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2slht\" (UniqueName: \"kubernetes.io/projected/61b96941-be95-4b6c-bd41-8d8cef2f4fd6-kube-api-access-2slht\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:04.995588 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.995551 4920 generic.go:356] "Generic (PLEG): container finished" podID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerID="84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee" exitCode=0 Oct 30 09:57:04.995776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.995608 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" Oct 30 09:57:04.995776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.995630 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" event={"ID":"61b96941-be95-4b6c-bd41-8d8cef2f4fd6","Type":"ContainerDied","Data":"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee"} Oct 30 09:57:04.995776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.995673 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" event={"ID":"61b96941-be95-4b6c-bd41-8d8cef2f4fd6","Type":"ContainerDied","Data":"48c4fce800c29257223cd7a2c3ca6246861ce61f8515ad3218a98d6ad77cece5"} Oct 30 09:57:04.995776 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:04.995694 4920 scope.go:122] "RemoveContainer" containerID="84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee" Oct 30 09:57:05.008133 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.008091 4920 scope.go:122] "RemoveContainer" containerID="84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee" Oct 30 09:57:05.008504 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:05.008475 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee\": container with ID starting with 84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee not found: ID does not exist" containerID="84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee" Oct 30 09:57:05.008632 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.008511 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee"} err="failed to get container status \"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee\": rpc error: code = NotFound desc = could not find container \"84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee\": container with ID starting with 84029b4ca999e00ea23cf17167da4b0112278c2d556a6ea2600e7ad7172535ee not found: ID does not exist" Oct 30 09:57:05.049907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.049870 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:57:05.061710 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.061675 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j"] Oct 30 09:57:05.337198 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.331960 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:05.346981 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.346853 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2f6ee2/test-model-2f6ee2-kserve-796dd788b7-88x6j" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerName="main" probeResult="failure" output="Get \"https://10.132.1.14:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:57:05.618207 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:05.618113 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:06.402607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:06.402573 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" path="/var/lib/kubelet/pods/61b96941-be95-4b6c-bd41-8d8cef2f4fd6/volumes" Oct 30 09:57:06.708627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:06.708540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:06.807656 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:06.807623 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:07.085812 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:07.085779 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:07.561159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:07.561124 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:07.665242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:07.661750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:08.859025 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:08.858996 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:10.062676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.062641 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:10.164238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.163864 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:10.413577 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.413465 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:10.414234 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.414192 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:57:10.414623 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.414596 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" containerID="cri-o://39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae" gracePeriod=30 Oct 30 09:57:10.623242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.614602 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:57:10.757358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.755470 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:10.887571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:10.887534 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:12.556733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:12.556698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:14.284738 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:14.284698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:15.506167 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:15.506134 4920 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5716962_c02e_4388_83b0_b11c9023bc90.slice/crio-conmon-39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:57:15.740847 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.740818 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:57:15.769298 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.769243 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" containerID="cri-o://490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07" gracePeriod=45 Oct 30 09:57:15.803006 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.802978 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803046 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803189 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803077 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803207 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803259 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803303 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803343 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:15.803500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803350 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803460 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803634 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\") pod \"d5716962-c02e-4388-83b0-b11c9023bc90\" (UID: \"d5716962-c02e-4388-83b0-b11c9023bc90\") " Oct 30 09:57:15.803720 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803673 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:15.804001 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.803967 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:15.804110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804003 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:15.804239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804201 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:15.804506 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804484 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5716962-c02e-4388-83b0-b11c9023bc90-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.804599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804511 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.804599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804525 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.804599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804538 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.804599 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.804552 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.806262 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.806230 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:15.806394 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.806368 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:15.806910 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.806888 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "kube-api-access-gjlrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:15.806992 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.806919 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token" pod "d5716962-c02e-4388-83b0-b11c9023bc90" (UID: "d5716962-c02e-4388-83b0-b11c9023bc90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:15.864697 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.864651 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:57:15.906467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.906427 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5716962-c02e-4388-83b0-b11c9023bc90-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.906618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.906472 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjlrv\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-kube-api-access-gjlrv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.906618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.906490 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5716962-c02e-4388-83b0-b11c9023bc90-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.906618 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.906505 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5716962-c02e-4388-83b0-b11c9023bc90-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:15.907759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.907722 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Oct 30 09:57:15.966132 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:15.966023 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:16.047603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.047577 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:16.056839 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.056813 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.114768 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.114899 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\" (UniqueName: \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.114973 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.115011 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.115075 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.124239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.115343 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\") pod \"6937d716-3d86-492b-b48a-42e96fbe40d4\" (UID: \"6937d716-3d86-492b-b48a-42e96fbe40d4\") " Oct 30 09:57:16.130463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.130416 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.131082 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.131056 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.148691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.146897 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:16.148691 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.147341 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.163402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.163354 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.167522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.167441 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.171020 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.170984 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf" pod "6937d716-3d86-492b-b48a-42e96fbe40d4" (UID: "6937d716-3d86-492b-b48a-42e96fbe40d4"). InnerVolumeSpecName "kube-api-access-7s8jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:16.172830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.172526 4920 generic.go:356] "Generic (PLEG): container finished" podID="d5716962-c02e-4388-83b0-b11c9023bc90" containerID="39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae" exitCode=0 Oct 30 09:57:16.172830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.172630 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerDied","Data":"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae"} Oct 30 09:57:16.172830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.172722 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" event={"ID":"d5716962-c02e-4388-83b0-b11c9023bc90","Type":"ContainerDied","Data":"847ea1c61be36984ed68642e8b6f869fe8d831fb23acb2c6f983e4b885bd57af"} Oct 30 09:57:16.172830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.172752 4920 scope.go:122] "RemoveContainer" containerID="39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae" Oct 30 09:57:16.173098 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.172753 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc" Oct 30 09:57:16.175301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.175273 4920 generic.go:356] "Generic (PLEG): container finished" podID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerID="490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07" exitCode=0 Oct 30 09:57:16.175420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.175331 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" event={"ID":"6937d716-3d86-492b-b48a-42e96fbe40d4","Type":"ContainerDied","Data":"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07"} Oct 30 09:57:16.175420 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.175365 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" event={"ID":"6937d716-3d86-492b-b48a-42e96fbe40d4","Type":"ContainerDied","Data":"e168d6e667d990a91dbfb05661563ba1c1e475fa849ead9771bcc2dfb055e3f4"} Oct 30 09:57:16.175517 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.175429 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2" Oct 30 09:57:16.189017 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.188991 4920 scope.go:122] "RemoveContainer" containerID="297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b" Oct 30 09:57:16.207180 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.207155 4920 scope.go:122] "RemoveContainer" containerID="39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae" Oct 30 09:57:16.207629 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:16.207572 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae\": container with ID starting with 39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae not found: ID does not exist" containerID="39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae" Oct 30 09:57:16.207629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.207612 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae"} err="failed to get container status \"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae\": rpc error: code = NotFound desc = could not find container \"39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae\": container with ID starting with 39197671fdba1e853440f6733349c8c6072575a1429497f406a5724988a4f2ae not found: ID does not exist" Oct 30 09:57:16.207806 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.207638 4920 scope.go:122] "RemoveContainer" containerID="297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b" Oct 30 09:57:16.208012 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:16.207987 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b\": container with ID starting with 297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b not found: ID does not exist" containerID="297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b" Oct 30 09:57:16.208070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.208021 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b"} err="failed to get container status \"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b\": rpc error: code = NotFound desc = could not find container \"297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b\": container with ID starting with 297c25a14cbf2a750937dc71db8d9d8d64d9268f9a6c1cbd2bed991442b8b23b not found: ID does not exist" Oct 30 09:57:16.208070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.208043 4920 scope.go:122] "RemoveContainer" containerID="490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07" Oct 30 09:57:16.224482 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.224378 4920 scope.go:122] "RemoveContainer" containerID="490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07" Oct 30 09:57:16.230749 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:16.230691 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07\": container with ID starting with 490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07 not found: ID does not exist" containerID="490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07" Oct 30 09:57:16.230904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.230761 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07"} err="failed to get container status \"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07\": rpc error: code = NotFound desc = could not find container \"490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07\": container with ID starting with 490140f0f44abb80afef31510e0815196b113f42935964150f418d5e169b1c07 not found: ID does not exist" Oct 30 09:57:16.234280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.233894 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.234280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.233930 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8jf\" (UniqueName: \"kubernetes.io/projected/6937d716-3d86-492b-b48a-42e96fbe40d4-kube-api-access-7s8jf\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.234280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.233944 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6937d716-3d86-492b-b48a-42e96fbe40d4-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.234280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.234104 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.234280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.234118 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6937d716-3d86-492b-b48a-42e96fbe40d4-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.241322 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.241297 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:57:16.245275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.245248 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/simulator-w2-e3d2ff-kserve-5cf99c47c9-rrnh2"] Oct 30 09:57:16.280081 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.279978 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:57:16.285894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.285864 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a213d-openshift-default-b7b455459-946hc"] Oct 30 09:57:16.401054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.401015 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" path="/var/lib/kubelet/pods/6937d716-3d86-492b-b48a-42e96fbe40d4/volumes" Oct 30 09:57:16.401716 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.401693 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" path="/var/lib/kubelet/pods/d5716962-c02e-4388-83b0-b11c9023bc90/volumes" Oct 30 09:57:16.532940 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.532833 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:16.648240 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:16.639242 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:17.644289 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:17.644253 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:18.876514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:18.876480 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:57:18.876998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:18.876722 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" podUID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" containerName="maas-api" containerID="cri-o://88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365" gracePeriod=30 Oct 30 09:57:18.934043 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:18.933996 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:19.307249 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.303754 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:19.406426 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.406394 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:19.508090 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.508060 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:19.543531 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.543504 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.733181 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\") pod \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.733269 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\") pod \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.733411 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\") pod \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.733482 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\") pod \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\" (UID: \"01c4c19f-4acd-4b1d-9212-58651e8d4a97\") " Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.735793 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca" pod "01c4c19f-4acd-4b1d-9212-58651e8d4a97" (UID: "01c4c19f-4acd-4b1d-9212-58651e8d4a97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:19.750299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.742209 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs" pod "01c4c19f-4acd-4b1d-9212-58651e8d4a97" (UID: "01c4c19f-4acd-4b1d-9212-58651e8d4a97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:19.819239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.807084 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x" pod "01c4c19f-4acd-4b1d-9212-58651e8d4a97" (UID: "01c4c19f-4acd-4b1d-9212-58651e8d4a97"). InnerVolumeSpecName "kube-api-access-gsh7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:19.819239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.818734 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls" pod "01c4c19f-4acd-4b1d-9212-58651e8d4a97" (UID: "01c4c19f-4acd-4b1d-9212-58651e8d4a97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:19.843251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.838101 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/01c4c19f-4acd-4b1d-9212-58651e8d4a97-kube-api-access-gsh7x\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.843251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.838143 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c4c19f-4acd-4b1d-9212-58651e8d4a97-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.843251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.838161 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.843251 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.838176 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c4c19f-4acd-4b1d-9212-58651e8d4a97-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.905675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:19.905642 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:20.007055 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.007013 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:20.107313 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.107279 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:20.261781 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.261742 4920 generic.go:356] "Generic (PLEG): container finished" podID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" containerID="88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365" exitCode=0 Oct 30 09:57:20.261982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.261790 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" event={"ID":"01c4c19f-4acd-4b1d-9212-58651e8d4a97","Type":"ContainerDied","Data":"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365"} Oct 30 09:57:20.261982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.261819 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" Oct 30 09:57:20.261982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.261852 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm" event={"ID":"01c4c19f-4acd-4b1d-9212-58651e8d4a97","Type":"ContainerDied","Data":"8bfa727c3fc3d324ea7afbec15cf973abdaca55bbb1277ab842eb7425d1f065a"} Oct 30 09:57:20.261982 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.261875 4920 scope.go:122] "RemoveContainer" containerID="88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365" Oct 30 09:57:20.274079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.274054 4920 scope.go:122] "RemoveContainer" containerID="88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365" Oct 30 09:57:20.274410 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:20.274381 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365\": container with ID starting with 88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365 not found: ID does not exist" containerID="88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365" Oct 30 09:57:20.274530 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.274418 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365"} err="failed to get container status \"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365\": rpc error: code = NotFound desc = could not find container \"88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365\": container with ID starting with 88280e47c866195c32e11b043ec3540833e76e30dc5b3acb5ce261cd988a1365 not found: ID does not exist" Oct 30 09:57:20.339698 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.339661 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:57:20.356432 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.356400 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f6ee2-69bccfd76c-dq7nm"] Oct 30 09:57:20.400872 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.400797 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" path="/var/lib/kubelet/pods/01c4c19f-4acd-4b1d-9212-58651e8d4a97/volumes" Oct 30 09:57:20.535798 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:20.535766 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:21.604979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:21.604941 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:57:21.711239 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:21.710704 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:21.788290 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:21.788256 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:23.339554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:23.339521 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:24.909095 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:24.909055 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:26.311974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.311932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:26.414329 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.414295 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:26.521759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.521704 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:26.608571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.605286 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" podUID="9b5212c0-889d-445c-ad41-7abf90a81555" containerName="payload-pre-processing" containerID="cri-o://d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827" gracePeriod=25 Oct 30 09:57:26.640861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.637521 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:57:26.640861 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.637839 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" containerID="cri-o://c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a" gracePeriod=30 Oct 30 09:57:26.650408 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:26.650009 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:27.067624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.067595 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:27.075754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.075733 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:57:27.187153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.186713 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\" (UniqueName: \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\") pod \"9b5212c0-889d-445c-ad41-7abf90a81555\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " Oct 30 09:57:27.187153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.187001 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\") pod \"9b5212c0-889d-445c-ad41-7abf90a81555\" (UID: \"9b5212c0-889d-445c-ad41-7abf90a81555\") " Oct 30 09:57:27.187913 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.187834 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:27.188076 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.188051 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume" pod "9b5212c0-889d-445c-ad41-7abf90a81555" (UID: "9b5212c0-889d-445c-ad41-7abf90a81555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:27.191421 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.191388 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw" pod "9b5212c0-889d-445c-ad41-7abf90a81555" (UID: "9b5212c0-889d-445c-ad41-7abf90a81555"). InnerVolumeSpecName "kube-api-access-s69rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:27.289541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.289506 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5212c0-889d-445c-ad41-7abf90a81555-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:27.289541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.289537 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s69rw\" (UniqueName: \"kubernetes.io/projected/9b5212c0-889d-445c-ad41-7abf90a81555-kube-api-access-s69rw\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:27.379117 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.379080 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:57:27.379652 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.379295 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" containerID="cri-o://b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b" gracePeriod=30 Oct 30 09:57:27.381907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.381874 4920 generic.go:356] "Generic (PLEG): container finished" podID="9b5212c0-889d-445c-ad41-7abf90a81555" containerID="d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827" exitCode=0 Oct 30 09:57:27.382060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.381945 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" Oct 30 09:57:27.382145 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.382124 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" event={"ID":"9b5212c0-889d-445c-ad41-7abf90a81555","Type":"ContainerDied","Data":"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827"} Oct 30 09:57:27.382355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.382203 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p" event={"ID":"9b5212c0-889d-445c-ad41-7abf90a81555","Type":"ContainerDied","Data":"85b9dd055f517e38ef746e0886c9fe2e45b1958c9457c99921ce18bd4a9a86b2"} Oct 30 09:57:27.382355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.382279 4920 scope.go:122] "RemoveContainer" containerID="d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827" Oct 30 09:57:27.411384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.411354 4920 scope.go:122] "RemoveContainer" containerID="d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827" Oct 30 09:57:27.411814 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:27.411789 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827\": container with ID starting with d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827 not found: ID does not exist" containerID="d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827" Oct 30 09:57:27.411924 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.411826 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827"} err="failed to get container status \"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827\": rpc error: code = NotFound desc = could not find container \"d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827\": container with ID starting with d3d0b7aaa3391bb32096b5b9841692cbb975a528dc3561d12e9fa294a1524827 not found: ID does not exist" Oct 30 09:57:27.422671 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.422618 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:27.457445 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.457411 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:57:27.460725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:27.460698 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-176b6c-755fcbcb8c-9695p"] Oct 30 09:57:28.284536 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:28.284499 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:28.400753 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:28.400716 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5212c0-889d-445c-ad41-7abf90a81555" path="/var/lib/kubelet/pods/9b5212c0-889d-445c-ad41-7abf90a81555/volumes" Oct 30 09:57:29.782906 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:29.782868 4920 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:57:29.783331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:29.782948 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:57:29.867039 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:29.867003 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:30.894931 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:30.894877 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" containerID="cri-o://c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f" gracePeriod=45 Oct 30 09:57:31.145290 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:31.145148 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:57:31.193282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.193257 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:57:31.254012 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.253977 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.254211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.254044 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\" (UniqueName: \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.254211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.254150 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.254211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.254181 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.254761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.254455 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.255534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.255302 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.255534 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.255313 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\") pod \"7cdc64f0-df08-435a-8583-81fed012ba0f\" (UID: \"7cdc64f0-df08-435a-8583-81fed012ba0f\") " Oct 30 09:57:31.255727 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.255542 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.256317 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.256289 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.256431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.256319 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.256431 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.256386 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.261197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.261157 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.261197 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.261157 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.261361 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.261234 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt" pod "7cdc64f0-df08-435a-8583-81fed012ba0f" (UID: "7cdc64f0-df08-435a-8583-81fed012ba0f"). InnerVolumeSpecName "kube-api-access-hlxnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.357702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.357663 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdc64f0-df08-435a-8583-81fed012ba0f-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.357702 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.357700 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.357934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.357713 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdc64f0-df08-435a-8583-81fed012ba0f-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.357934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.357727 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxnt\" (UniqueName: \"kubernetes.io/projected/7cdc64f0-df08-435a-8583-81fed012ba0f-kube-api-access-hlxnt\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.421083 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.420991 4920 generic.go:356] "Generic (PLEG): container finished" podID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerID="c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f" exitCode=0 Oct 30 09:57:31.421349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.421257 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" event={"ID":"7cdc64f0-df08-435a-8583-81fed012ba0f","Type":"ContainerDied","Data":"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f"} Oct 30 09:57:31.421349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.421301 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" event={"ID":"7cdc64f0-df08-435a-8583-81fed012ba0f","Type":"ContainerDied","Data":"138dfa6f940500b478a3c22a9a3bc34aca44ad58fafe486affc8a02eea19435a"} Oct 30 09:57:31.421349 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.421326 4920 scope.go:122] "RemoveContainer" containerID="c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f" Oct 30 09:57:31.421562 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.421131 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424" Oct 30 09:57:31.436261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.436196 4920 scope.go:122] "RemoveContainer" containerID="c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f" Oct 30 09:57:31.436611 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:31.436594 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f\": container with ID starting with c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f not found: ID does not exist" containerID="c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f" Oct 30 09:57:31.436659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.436621 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f"} err="failed to get container status \"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f\": rpc error: code = NotFound desc = could not find container \"c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f\": container with ID starting with c79c62024e9446bf50056b0f5e5936f1e0662c539d1d4e72c10536ca12ca623f not found: ID does not exist" Oct 30 09:57:31.477016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.476982 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:57:31.480376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.480347 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-2-w3-176b6c-kserve-cb49598bb-mb424"] Oct 30 09:57:31.550430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.550384 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:31.939253 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.939189 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:57:31.966434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966400 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.966642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966480 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.966642 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966610 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.966787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966640 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.966787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966683 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.966787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966687 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.966787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966751 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.967010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966782 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.967010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966837 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.967010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966875 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.967010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966940 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\") pod \"aafe5514-75c3-4add-95bb-92b5759017a9\" (UID: \"aafe5514-75c3-4add-95bb-92b5759017a9\") " Oct 30 09:57:31.967010 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.966992 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.967296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967067 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.967378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967353 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:31.967561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967529 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.967561 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967553 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.967700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967568 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.967700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967583 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafe5514-75c3-4add-95bb-92b5759017a9-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.967700 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.967596 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.968966 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.968936 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:31.969241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.969185 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.969439 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.969422 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "kube-api-access-kctml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.969514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:31.969450 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token" pod "aafe5514-75c3-4add-95bb-92b5759017a9" (UID: "aafe5514-75c3-4add-95bb-92b5759017a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:32.068820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.068785 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafe5514-75c3-4add-95bb-92b5759017a9-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.068820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.068816 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.068820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.068828 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafe5514-75c3-4add-95bb-92b5759017a9-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.069079 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.068838 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kctml\" (UniqueName: \"kubernetes.io/projected/aafe5514-75c3-4add-95bb-92b5759017a9-kube-api-access-kctml\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.399788 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.399751 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" path="/var/lib/kubelet/pods/7cdc64f0-df08-435a-8583-81fed012ba0f/volumes" Oct 30 09:57:32.430548 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.430511 4920 generic.go:356] "Generic (PLEG): container finished" podID="aafe5514-75c3-4add-95bb-92b5759017a9" containerID="c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a" exitCode=0 Oct 30 09:57:32.430807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.430578 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" Oct 30 09:57:32.430807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.430594 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerDied","Data":"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a"} Oct 30 09:57:32.430807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.430660 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp" event={"ID":"aafe5514-75c3-4add-95bb-92b5759017a9","Type":"ContainerDied","Data":"445fcd2f492bdea4df72901af37d259b365c2eb6efe0c72b941f4c9f0cfd99f1"} Oct 30 09:57:32.430807 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.430691 4920 scope.go:122] "RemoveContainer" containerID="c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a" Oct 30 09:57:32.495865 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.495840 4920 scope.go:122] "RemoveContainer" containerID="866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b" Oct 30 09:57:32.507296 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.507271 4920 scope.go:122] "RemoveContainer" containerID="c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a" Oct 30 09:57:32.507666 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:32.507642 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a\": container with ID starting with c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a not found: ID does not exist" containerID="c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a" Oct 30 09:57:32.507887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.507677 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a"} err="failed to get container status \"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a\": rpc error: code = NotFound desc = could not find container \"c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a\": container with ID starting with c46f15fbccd49b2600f632ccc8c9d52d96f5f5f25b43b379085c645b7c50580a not found: ID does not exist" Oct 30 09:57:32.507887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.507703 4920 scope.go:122] "RemoveContainer" containerID="866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b" Oct 30 09:57:32.508071 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:32.508050 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b\": container with ID starting with 866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b not found: ID does not exist" containerID="866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b" Oct 30 09:57:32.508147 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.508081 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b"} err="failed to get container status \"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b\": rpc error: code = NotFound desc = could not find container \"866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b\": container with ID starting with 866c061f67da374770142ce40f99a02c8b113ae404454f547b5f560df8d42a7b not found: ID does not exist" Oct 30 09:57:32.511885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.511848 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:57:32.522641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.522610 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f6ee2-openshift-default-5c8958659f-xnjrp"] Oct 30 09:57:32.643448 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.643419 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:57:32.678942 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.678803 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679212 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679452 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679563 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679628 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679619 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679761 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679742 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679843 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679828 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679886 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.679929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.679911 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\") pod \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\" (UID: \"f7b41fc0-f9a2-4cb3-bd10-74700220b99c\") " Oct 30 09:57:32.681070 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.680688 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.682449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.681909 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.682449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.682241 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.682449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.682263 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.682979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.682753 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.683066 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.682996 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.684061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.684022 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:32.690151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.690093 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "kube-api-access-vlwnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:32.756714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.739264 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:32.756714 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.746398 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:32.774241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.759286 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy" pod "f7b41fc0-f9a2-4cb3-bd10-74700220b99c" (UID: "f7b41fc0-f9a2-4cb3-bd10-74700220b99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800145 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800184 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800199 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800231 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800247 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800259 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.823659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:32.800271 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlwnd\" (UniqueName: \"kubernetes.io/projected/f7b41fc0-f9a2-4cb3-bd10-74700220b99c-kube-api-access-vlwnd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:33.258414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.258371 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:33.442772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.442735 4920 generic.go:356] "Generic (PLEG): container finished" podID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerID="b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b" exitCode=0 Oct 30 09:57:33.442984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.442786 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" Oct 30 09:57:33.442984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.442810 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" event={"ID":"f7b41fc0-f9a2-4cb3-bd10-74700220b99c","Type":"ContainerDied","Data":"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b"} Oct 30 09:57:33.442984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.442863 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9" event={"ID":"f7b41fc0-f9a2-4cb3-bd10-74700220b99c","Type":"ContainerDied","Data":"4fb43df0408a779fc818faf23e7b3a9b46de1f022ceecf6840a4a913ee12f852"} Oct 30 09:57:33.442984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.442884 4920 scope.go:122] "RemoveContainer" containerID="b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b" Oct 30 09:57:33.453929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.453908 4920 scope.go:122] "RemoveContainer" containerID="b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b" Oct 30 09:57:33.454271 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:57:33.454247 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b\": container with ID starting with b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b not found: ID does not exist" containerID="b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b" Oct 30 09:57:33.454364 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.454280 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b"} err="failed to get container status \"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b\": rpc error: code = NotFound desc = could not find container \"b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b\": container with ID starting with b37bd9ccb0cd0ddaee7f396a73bc5e5ab8747da89518d29a02a583a8a2a0516b not found: ID does not exist" Oct 30 09:57:33.509399 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.509279 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:57:33.521276 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.521200 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dzpl9"] Oct 30 09:57:33.767159 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.767072 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:33.870092 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:33.870062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:34.398673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:34.398642 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" path="/var/lib/kubelet/pods/aafe5514-75c3-4add-95bb-92b5759017a9/volumes" Oct 30 09:57:34.399179 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:34.399163 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" path="/var/lib/kubelet/pods/f7b41fc0-f9a2-4cb3-bd10-74700220b99c/volumes" Oct 30 09:57:34.875250 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:34.875203 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:36.363108 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:36.363049 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:37.579353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:37.579303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:37.883778 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:37.882005 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:37.977035 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:37.977003 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:38.174950 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:38.174855 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:38.291813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:38.291783 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:38.394946 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:38.394870 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:38.527303 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:38.521400 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:39.524099 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:39.524062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:40.056341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:40.056271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:40.148679 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:40.148647 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:40.227038 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:40.226998 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:40.721060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:40.721021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:42.008284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:42.008244 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:43.703715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:43.703678 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:45.141131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:45.141098 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:45.258175 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:45.258132 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:45.418546 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:45.418459 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:47.033943 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:47.033908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:47.133922 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:47.133887 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:47.465187 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:47.465139 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:48.355600 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.355564 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:57:48.356121 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.355791 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" podUID="7b9de14d-2ed8-4644-806f-f034ed71b659" containerName="maas-api" containerID="cri-o://77822ef47a6ca338b3c7943efafac762ec40a8d84ad62871c63fe1a5cdb8f6b7" gracePeriod=30 Oct 30 09:57:48.412919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.412878 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:48.884537 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.884495 4920 generic.go:356] "Generic (PLEG): container finished" podID="7b9de14d-2ed8-4644-806f-f034ed71b659" containerID="77822ef47a6ca338b3c7943efafac762ec40a8d84ad62871c63fe1a5cdb8f6b7" exitCode=0 Oct 30 09:57:48.884963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.884943 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" event={"ID":"7b9de14d-2ed8-4644-806f-f034ed71b659","Type":"ContainerDied","Data":"77822ef47a6ca338b3c7943efafac762ec40a8d84ad62871c63fe1a5cdb8f6b7"} Oct 30 09:57:48.905050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.905018 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:48.912844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:48.912817 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:57:49.003428 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.003388 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\") pod \"7b9de14d-2ed8-4644-806f-f034ed71b659\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " Oct 30 09:57:49.003612 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.003451 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\" (UniqueName: \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\") pod \"7b9de14d-2ed8-4644-806f-f034ed71b659\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " Oct 30 09:57:49.003659 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.003625 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\") pod \"7b9de14d-2ed8-4644-806f-f034ed71b659\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " Oct 30 09:57:49.003709 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.003661 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\") pod \"7b9de14d-2ed8-4644-806f-f034ed71b659\" (UID: \"7b9de14d-2ed8-4644-806f-f034ed71b659\") " Oct 30 09:57:49.003997 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.003967 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca" pod "7b9de14d-2ed8-4644-806f-f034ed71b659" (UID: "7b9de14d-2ed8-4644-806f-f034ed71b659"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:49.004316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.004280 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b9de14d-2ed8-4644-806f-f034ed71b659-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:49.006042 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.006015 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:49.006745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.006306 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls" pod "7b9de14d-2ed8-4644-806f-f034ed71b659" (UID: "7b9de14d-2ed8-4644-806f-f034ed71b659"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:49.006745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.006363 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2" pod "7b9de14d-2ed8-4644-806f-f034ed71b659" (UID: "7b9de14d-2ed8-4644-806f-f034ed71b659"). InnerVolumeSpecName "kube-api-access-gkpf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:49.006745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.006626 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs" pod "7b9de14d-2ed8-4644-806f-f034ed71b659" (UID: "7b9de14d-2ed8-4644-806f-f034ed71b659"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:49.088610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.088572 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:49.105920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.105886 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkpf2\" (UniqueName: \"kubernetes.io/projected/7b9de14d-2ed8-4644-806f-f034ed71b659-kube-api-access-gkpf2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:49.105920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.105917 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:49.105920 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.105927 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b9de14d-2ed8-4644-806f-f034ed71b659-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:57:49.573564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.573526 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:49.893128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.893049 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" event={"ID":"7b9de14d-2ed8-4644-806f-f034ed71b659","Type":"ContainerDied","Data":"d22024d6078b66f996380914d874ba372383a90e12261b0da174ac2b746805b1"} Oct 30 09:57:49.893128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.893088 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j" Oct 30 09:57:49.893128 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.893109 4920 scope.go:122] "RemoveContainer" containerID="77822ef47a6ca338b3c7943efafac762ec40a8d84ad62871c63fe1a5cdb8f6b7" Oct 30 09:57:49.942929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.942892 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:57:49.954631 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:49.954601 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bdd208-7d64b95fd-lf47j"] Oct 30 09:57:50.396715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:50.396667 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:50.402190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:50.402159 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9de14d-2ed8-4644-806f-f034ed71b659" path="/var/lib/kubelet/pods/7b9de14d-2ed8-4644-806f-f034ed71b659/volumes" Oct 30 09:57:50.834274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:50.825812 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:50.970403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:50.970366 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:51.075952 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:51.075915 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:51.184845 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:51.177872 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:51.607759 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:51.607726 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:52.655729 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:52.655698 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:53.065002 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:53.064962 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:53.209275 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:53.208948 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:53.304430 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:53.304396 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:54.124508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:54.124471 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:54.444295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:54.438680 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:54.537311 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:54.537280 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:55.590915 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:55.590881 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:55.671963 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:55.671926 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:55.932574 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:55.932445 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:57:55.933062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:55.933024 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" containerID="cri-o://9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c" gracePeriod=30 Oct 30 09:57:55.947304 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:55.947025 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:56.086238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:56.073803 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:57.629903 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:57.629869 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:57.989404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:57.989313 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:58.089624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.089589 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:58.214693 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.214649 4920 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:57:58.214911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.214730 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:57:58.327335 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.327299 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:57:58.327860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.327793 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" containerID="cri-o://bce1e3add6476bb7b78ef2fcca2c1253c05f11540b14843180aa326b1899d620" gracePeriod=30 Oct 30 09:57:58.349610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:58.349565 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:59.372573 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:59.372540 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:57:59.392813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:59.392778 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:57:59.985550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:57:59.985506 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:01.038299 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.038264 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:01.172727 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:58:01.172685 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:58:01.253622 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.253596 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:58:01.339493 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339456 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339505 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339544 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339575 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339624 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339660 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339731 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.339974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339781 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.339974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339835 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.339974 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339850 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.340188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.339959 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.340188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340022 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\") pod \"ccea84a3-36c1-4d20-a018-59db9ed75f16\" (UID: \"ccea84a3-36c1-4d20-a018-59db9ed75f16\") " Oct 30 09:58:01.340188 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340107 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:01.340377 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340347 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.340607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340528 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.340607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340551 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.340607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340564 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.340607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340573 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.340607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.340582 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccea84a3-36c1-4d20-a018-59db9ed75f16-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.342071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.342048 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "kube-api-access-22gpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:01.342157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.342050 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:01.342379 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.342357 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.342468 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.342363 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token" pod "ccea84a3-36c1-4d20-a018-59db9ed75f16" (UID: "ccea84a3-36c1-4d20-a018-59db9ed75f16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:01.441654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.441611 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gpk\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-kube-api-access-22gpk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.441654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.441644 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.441654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.441654 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.441654 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:01.441662 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccea84a3-36c1-4d20-a018-59db9ed75f16-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:02.052547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.052511 4920 generic.go:356] "Generic (PLEG): container finished" podID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerID="9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c" exitCode=0 Oct 30 09:58:02.053013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.052560 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" Oct 30 09:58:02.053013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.052598 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerDied","Data":"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c"} Oct 30 09:58:02.053013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.052658 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b" event={"ID":"ccea84a3-36c1-4d20-a018-59db9ed75f16","Type":"ContainerDied","Data":"fe1c9b285d193e101b90f3a8e800580d5349b23cd109421bb5129d6fe562ee82"} Oct 30 09:58:02.053013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.052686 4920 scope.go:122] "RemoveContainer" containerID="9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c" Oct 30 09:58:02.065995 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.065973 4920 scope.go:122] "RemoveContainer" containerID="9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7" Oct 30 09:58:02.076153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.076128 4920 scope.go:122] "RemoveContainer" containerID="9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c" Oct 30 09:58:02.076504 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:58:02.076479 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c\": container with ID starting with 9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c not found: ID does not exist" containerID="9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c" Oct 30 09:58:02.076605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.076513 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c"} err="failed to get container status \"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c\": rpc error: code = NotFound desc = could not find container \"9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c\": container with ID starting with 9b2f72d052ae0732b422626a51149f7414f7fd61e4f55b671707534f88ba453c not found: ID does not exist" Oct 30 09:58:02.076605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.076537 4920 scope.go:122] "RemoveContainer" containerID="9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7" Oct 30 09:58:02.076815 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:58:02.076795 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7\": container with ID starting with 9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7 not found: ID does not exist" containerID="9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7" Oct 30 09:58:02.076891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.076822 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7"} err="failed to get container status \"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7\": rpc error: code = NotFound desc = could not find container \"9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7\": container with ID starting with 9f0218501626a9d82f9490d6ec66da81e67ae37a702aab5d1c09032aae472dc7 not found: ID does not exist" Oct 30 09:58:02.114585 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.114551 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:58:02.124971 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.124939 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-cnr9b"] Oct 30 09:58:02.398960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.398867 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" path="/var/lib/kubelet/pods/ccea84a3-36c1-4d20-a018-59db9ed75f16/volumes" Oct 30 09:58:02.654119 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:02.654038 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:03.602051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.601987 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:58:03.670048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670013 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670105 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670177 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670301 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670202 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670341 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670383 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.670514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670410 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670514 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670477 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670535 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.670756 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.670565 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\") pod \"e38f1e38-8109-41b7-84a1-8b83fa173850\" (UID: \"e38f1e38-8109-41b7-84a1-8b83fa173850\") " Oct 30 09:58:03.671089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.671064 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:03.671089 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.671083 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.671287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.671130 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.671287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.671164 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.671287 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.671189 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.673592 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.673550 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:03.674495 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.674468 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.674636 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.674494 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:03.674773 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.674754 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9" pod "e38f1e38-8109-41b7-84a1-8b83fa173850" (UID: "e38f1e38-8109-41b7-84a1-8b83fa173850"). InnerVolumeSpecName "kube-api-access-58qf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:03.772619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772585 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772619 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772620 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772632 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772644 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38f1e38-8109-41b7-84a1-8b83fa173850-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772657 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qf9\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-kube-api-access-58qf9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772671 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38f1e38-8109-41b7-84a1-8b83fa173850-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772684 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.772828 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.772696 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38f1e38-8109-41b7-84a1-8b83fa173850-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.915211 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:03.915112 4920 scope.go:122] "RemoveContainer" containerID="bce1e3add6476bb7b78ef2fcca2c1253c05f11540b14843180aa326b1899d620" Oct 30 09:58:04.069741 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.069703 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" event={"ID":"e38f1e38-8109-41b7-84a1-8b83fa173850","Type":"ContainerDied","Data":"bce1e3add6476bb7b78ef2fcca2c1253c05f11540b14843180aa326b1899d620"} Oct 30 09:58:04.069900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.069760 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" event={"ID":"e38f1e38-8109-41b7-84a1-8b83fa173850","Type":"ContainerDied","Data":"c4c96db62f5a495c167f92ea292f9f9ba7ce1b01d7cbb5f0a6c1a21d449723a0"} Oct 30 09:58:04.069900 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.069798 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq" Oct 30 09:58:04.147749 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.147713 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:58:04.150929 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.150903 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-m8lhq"] Oct 30 09:58:04.154580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.154553 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:04.398549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:04.398513 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" path="/var/lib/kubelet/pods/e38f1e38-8109-41b7-84a1-8b83fa173850/volumes" Oct 30 09:58:05.611401 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:05.611350 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:07.090483 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:07.090445 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:07.193143 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:07.193110 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:07.311273 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:07.311211 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:08.626268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:08.626234 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:09.921685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:09.921649 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:58:09.922131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:09.921915 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" podUID="094edb06-2aff-4715-8a39-86be7ed5746f" containerName="maas-api" containerID="cri-o://71f584ede9a414324b9c7678327a61741397a75c241b6ace2b0f547beff4d81b" gracePeriod=30 Oct 30 09:58:09.972182 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:09.972116 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:10.119302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.119263 4920 generic.go:356] "Generic (PLEG): container finished" podID="094edb06-2aff-4715-8a39-86be7ed5746f" containerID="71f584ede9a414324b9c7678327a61741397a75c241b6ace2b0f547beff4d81b" exitCode=0 Oct 30 09:58:10.119481 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.119425 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" event={"ID":"094edb06-2aff-4715-8a39-86be7ed5746f","Type":"ContainerDied","Data":"71f584ede9a414324b9c7678327a61741397a75c241b6ace2b0f547beff4d81b"} Oct 30 09:58:10.429131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.424637 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:58:10.435429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.435391 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:10.516576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.516537 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\") pod \"094edb06-2aff-4715-8a39-86be7ed5746f\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " Oct 30 09:58:10.516782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.516682 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\") pod \"094edb06-2aff-4715-8a39-86be7ed5746f\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " Oct 30 09:58:10.516782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.516737 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\") pod \"094edb06-2aff-4715-8a39-86be7ed5746f\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " Oct 30 09:58:10.516951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.516933 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\" (UniqueName: \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\") pod \"094edb06-2aff-4715-8a39-86be7ed5746f\" (UID: \"094edb06-2aff-4715-8a39-86be7ed5746f\") " Oct 30 09:58:10.517044 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.517015 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca" pod "094edb06-2aff-4715-8a39-86be7ed5746f" (UID: "094edb06-2aff-4715-8a39-86be7ed5746f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:10.517491 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.517465 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/094edb06-2aff-4715-8a39-86be7ed5746f-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:10.519746 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.519717 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs" pod "094edb06-2aff-4715-8a39-86be7ed5746f" (UID: "094edb06-2aff-4715-8a39-86be7ed5746f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:10.519850 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.519807 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls" pod "094edb06-2aff-4715-8a39-86be7ed5746f" (UID: "094edb06-2aff-4715-8a39-86be7ed5746f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:10.520316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.520293 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq" pod "094edb06-2aff-4715-8a39-86be7ed5746f" (UID: "094edb06-2aff-4715-8a39-86be7ed5746f"). InnerVolumeSpecName "kube-api-access-v56lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:10.535028 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.534989 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:10.618547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.618454 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56lq\" (UniqueName: \"kubernetes.io/projected/094edb06-2aff-4715-8a39-86be7ed5746f-kube-api-access-v56lq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:10.618547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.618484 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:10.618547 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.618497 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/094edb06-2aff-4715-8a39-86be7ed5746f-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:10.658300 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:10.656631 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:11.063160 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.063123 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:11.127848 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.127815 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" Oct 30 09:58:11.128071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.127843 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws" event={"ID":"094edb06-2aff-4715-8a39-86be7ed5746f","Type":"ContainerDied","Data":"cb7a3d59a08cc58bf660e430e535d016496205bc3cb816669b95aadfa682419b"} Oct 30 09:58:11.128071 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.127914 4920 scope.go:122] "RemoveContainer" containerID="71f584ede9a414324b9c7678327a61741397a75c241b6ace2b0f547beff4d81b" Oct 30 09:58:11.177894 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.177820 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:58:11.181846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:11.181820 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae795b-57cc65798-bkbws"] Oct 30 09:58:12.068967 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:12.068929 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:12.398754 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:12.398667 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094edb06-2aff-4715-8a39-86be7ed5746f" path="/var/lib/kubelet/pods/094edb06-2aff-4715-8a39-86be7ed5746f/volumes" Oct 30 09:58:13.650418 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:13.648745 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:15.056153 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:15.056114 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:16.748544 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:16.748510 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:16.772970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:16.772934 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:58:16.850859 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:16.850820 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:17.111512 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:17.111469 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:58:17.111884 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:17.111850 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" containerID="cri-o://be6ebb8670f279cbf4805f4bdd0e79568bea12fc30f300132b681c5c42b8ea47" gracePeriod=30 Oct 30 09:58:17.125485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:17.125439 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:18.461186 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:18.461136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:20.148904 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:20.148863 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:21.605316 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:21.605271 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:22.310341 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.310308 4920 generic.go:356] "Generic (PLEG): container finished" podID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerID="be6ebb8670f279cbf4805f4bdd0e79568bea12fc30f300132b681c5c42b8ea47" exitCode=0 Oct 30 09:58:22.310554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.310456 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerDied","Data":"be6ebb8670f279cbf4805f4bdd0e79568bea12fc30f300132b681c5c42b8ea47"} Oct 30 09:58:22.310554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.310538 4920 scope.go:122] "RemoveContainer" containerID="62621ad9ea08776842807305264cd1b92e99e891d9dcc44a579b141bc1b1cc9c" Oct 30 09:58:22.393805 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.393727 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:58:22.495970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.495934 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496024 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496093 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496145 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496190 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496273 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496331 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496374 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.496660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.496447 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\") pod \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\" (UID: \"765b68b6-9f92-4c9b-b295-dafb87c8ab4e\") " Oct 30 09:58:22.497122 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.497089 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:22.497295 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.497127 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:22.497384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.497319 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:22.497384 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.497332 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:22.497545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.497523 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:22.498769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.498583 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:22.499200 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.499165 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:22.499616 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.499596 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:22.499914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.499892 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k" pod "765b68b6-9f92-4c9b-b295-dafb87c8ab4e" (UID: "765b68b6-9f92-4c9b-b295-dafb87c8ab4e"). InnerVolumeSpecName "kube-api-access-xnk8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:22.598390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598349 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598388 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598404 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnk8k\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-kube-api-access-xnk8k\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598419 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598432 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598445 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598456 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598469 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:22.598607 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:22.598482 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/765b68b6-9f92-4c9b-b295-dafb87c8ab4e-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:58:23.046996 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.046962 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:23.320675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.320594 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" Oct 30 09:58:23.320675 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.320633 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" event={"ID":"765b68b6-9f92-4c9b-b295-dafb87c8ab4e","Type":"ContainerDied","Data":"1502a9739b281388d76ee48a888b6b112c05097c38ee3ab2cd5cb321c7c462ae"} Oct 30 09:58:23.320886 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.320703 4920 scope.go:122] "RemoveContainer" containerID="be6ebb8670f279cbf4805f4bdd0e79568bea12fc30f300132b681c5c42b8ea47" Oct 30 09:58:23.367508 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.367472 4920 patch_prober.go:29] interesting pod/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:58:23.367718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.367529 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:58:23.384352 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.384318 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:58:23.398201 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:23.398162 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae795b-openshift-default-5b55749f85-xj7th"] Oct 30 09:58:24.398252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:24.398177 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" path="/var/lib/kubelet/pods/765b68b6-9f92-4c9b-b295-dafb87c8ab4e/volumes" Oct 30 09:58:24.664446 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:24.664361 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:26.146248 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:26.146180 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:27.638603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:27.638562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:28.259555 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:28.259514 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:28.359271 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:28.359231 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:29.130852 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:29.130813 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:30.742484 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:30.742449 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:31.137143 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:58:31.137102 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:58:32.100695 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:32.100656 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:32.596986 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:32.596929 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:32.698016 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:32.697980 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:32.804242 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:32.803879 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:33.228149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:33.228104 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:34.373615 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:34.373578 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:35.911854 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:35.911808 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:36.697736 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:36.697699 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:37.556779 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:37.556733 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:37.979292 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:37.969966 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:58:39.170064 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:39.170019 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:39.923505 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:39.923470 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:40.021279 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:40.021242 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:40.246485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:40.246392 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:40.754390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:40.754356 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:42.521457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:42.521422 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:44.028144 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:44.028099 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:45.563278 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:45.563238 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:47.170775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:47.170730 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:48.025787 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:48.025753 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:48.851641 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:48.851583 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:50.441715 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:50.441665 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:51.434804 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:51.434764 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:51.547456 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:51.547349 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:58:51.570241 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:51.550639 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:51.987102 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:51.987064 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:53.472704 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:53.472657 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:54.498568 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:54.498516 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:54.919091 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:54.919045 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:56.396750 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:56.396720 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:57.853620 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:57.853581 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:58.865195 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:58.865160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:58.964603 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:58.964569 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:58:59.356832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:58:59.356793 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:00.633831 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:00.633791 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:01.128844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:01.128757 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:01.137540 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:01.137504 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:59:01.230123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:01.230080 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:01.317629 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:01.317594 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:01.769857 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:01.769810 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:02.859060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:02.859026 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:03.881838 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:03.881809 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:59:03.889769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:03.889741 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 09:59:04.346627 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:04.346582 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:05.857136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:05.857102 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:07.352522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:07.352490 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:07.774510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:07.774418 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:07.875760 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:07.875718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:08.130449 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:08.130409 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:09.078183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:09.078148 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:09.104775 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:09.104742 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:59:10.536733 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:10.536700 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:11.116980 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:11.116946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:11.264504 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:11.264468 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:11.369860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:11.369771 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:11.751373 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:11.751275 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:13.055554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:13.055515 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:14.654613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:14.654577 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:16.161564 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:16.161531 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:17.666888 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:17.666850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:18.011259 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.011158 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:18.112021 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.111983 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:18.333909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.333873 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:59:18.335692 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.334133 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" containerID="cri-o://c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6" gracePeriod=30 Oct 30 09:59:18.346802 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.346773 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:59:18.347263 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.347185 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" containerID="cri-o://29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102" gracePeriod=30 Oct 30 09:59:18.356743 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:18.356719 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:19.319404 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:19.319369 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:19.778048 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:19.777951 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:19.899876 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:19.899836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:20.679529 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:20.679489 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:59:20.680032 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:20.679749 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" podUID="000155aa-ce40-4e39-a5b6-6961b01e88d1" containerName="maas-api" containerID="cri-o://285413df0db25c7952a2d1046055c7e41369e18849061868073fcb915b8c5f82" gracePeriod=30 Oct 30 09:59:20.756314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:20.756197 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:20.852723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:20.852675 4920 generic.go:356] "Generic (PLEG): container finished" podID="000155aa-ce40-4e39-a5b6-6961b01e88d1" containerID="285413df0db25c7952a2d1046055c7e41369e18849061868073fcb915b8c5f82" exitCode=0 Oct 30 09:59:20.852723 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:20.852710 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" event={"ID":"000155aa-ce40-4e39-a5b6-6961b01e88d1","Type":"ContainerDied","Data":"285413df0db25c7952a2d1046055c7e41369e18849061868073fcb915b8c5f82"} Oct 30 09:59:21.086247 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.084315 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:59:21.113161 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113131 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\") pod \"000155aa-ce40-4e39-a5b6-6961b01e88d1\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " Oct 30 09:59:21.113414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113186 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\" (UniqueName: \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\") pod \"000155aa-ce40-4e39-a5b6-6961b01e88d1\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " Oct 30 09:59:21.113414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113256 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\") pod \"000155aa-ce40-4e39-a5b6-6961b01e88d1\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " Oct 30 09:59:21.113414 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113322 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\") pod \"000155aa-ce40-4e39-a5b6-6961b01e88d1\" (UID: \"000155aa-ce40-4e39-a5b6-6961b01e88d1\") " Oct 30 09:59:21.113597 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113558 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca" pod "000155aa-ce40-4e39-a5b6-6961b01e88d1" (UID: "000155aa-ce40-4e39-a5b6-6961b01e88d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:21.113826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.113800 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000155aa-ce40-4e39-a5b6-6961b01e88d1-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:21.115545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.115519 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk" pod "000155aa-ce40-4e39-a5b6-6961b01e88d1" (UID: "000155aa-ce40-4e39-a5b6-6961b01e88d1"). InnerVolumeSpecName "kube-api-access-dgvdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:21.115647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.115579 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls" pod "000155aa-ce40-4e39-a5b6-6961b01e88d1" (UID: "000155aa-ce40-4e39-a5b6-6961b01e88d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:21.115647 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.115592 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs" pod "000155aa-ce40-4e39-a5b6-6961b01e88d1" (UID: "000155aa-ce40-4e39-a5b6-6961b01e88d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:21.214533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.214496 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:21.214725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.214586 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvdk\" (UniqueName: \"kubernetes.io/projected/000155aa-ce40-4e39-a5b6-6961b01e88d1-kube-api-access-dgvdk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:21.214725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.214606 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:21.214725 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.214615 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000155aa-ce40-4e39-a5b6-6961b01e88d1-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:21.315919 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.315877 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:21.398970 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.398938 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:21.851975 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.851939 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:21.862130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.862097 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" Oct 30 09:59:21.862130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.862124 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8" event={"ID":"000155aa-ce40-4e39-a5b6-6961b01e88d1","Type":"ContainerDied","Data":"4cdf33730b78052f15d8cada345eac4879c6f6415e2e449c892dd4b879f0dfa1"} Oct 30 09:59:21.862402 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.862188 4920 scope.go:122] "RemoveContainer" containerID="285413df0db25c7952a2d1046055c7e41369e18849061868073fcb915b8c5f82" Oct 30 09:59:21.918454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.918406 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:59:21.928168 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:21.928133 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c88ae-5c7b699988-7f8s8"] Oct 30 09:59:22.398151 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:22.398111 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000155aa-ce40-4e39-a5b6-6961b01e88d1" path="/var/lib/kubelet/pods/000155aa-ce40-4e39-a5b6-6961b01e88d1/volumes" Oct 30 09:59:23.018124 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.018091 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:23.636685 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.636652 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:59:23.650724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650680 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.650891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650738 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.650891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650782 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.650891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650815 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.650891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650846 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.650891 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650873 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.651205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650927 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.651205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.650965 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.651205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.651036 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\") pod \"3528fc4f-2bf6-499b-965a-a1738c881609\" (UID: \"3528fc4f-2bf6-499b-965a-a1738c881609\") " Oct 30 09:59:23.651415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.651393 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:23.651554 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.651537 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3528fc4f-2bf6-499b-965a-a1738c881609-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.651745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.651679 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.651877 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.651768 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.653060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.652788 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.653060 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.652843 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.654403 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.654377 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.654638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.654615 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "kube-api-access-r4kk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.659277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.655638 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:23.659277 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.655770 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token" pod "3528fc4f-2bf6-499b-965a-a1738c881609" (UID: "3528fc4f-2bf6-499b-965a-a1738c881609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.680766 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.680733 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:59:23.752415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752375 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752437 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752536 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752617 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752571 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752631 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752681 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752704 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.752813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752755 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.752813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752783 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.753112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.752897 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\") pod \"4e9c5ac0-342e-4fca-838f-f2871af96e55\" (UID: \"4e9c5ac0-342e-4fca-838f-f2871af96e55\") " Oct 30 09:59:23.753338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753248 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.753465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753407 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753413 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.753465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753431 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753446 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3528fc4f-2bf6-499b-965a-a1738c881609-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753465 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753460 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753474 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753487 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753500 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753512 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kk2\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-kube-api-access-r4kk2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753527 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3528fc4f-2bf6-499b-965a-a1738c881609-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.753624 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.753542 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3528fc4f-2bf6-499b-965a-a1738c881609-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.754063 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.754034 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.754150 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.754086 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:23.755369 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.755336 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.756274 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.756253 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:23.756480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.756452 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.756575 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.756488 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx" pod "4e9c5ac0-342e-4fca-838f-f2871af96e55" (UID: "4e9c5ac0-342e-4fca-838f-f2871af96e55"). InnerVolumeSpecName "kube-api-access-mdrvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.854155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854116 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854149 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854159 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854155 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854167 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9c5ac0-342e-4fca-838f-f2871af96e55-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854176 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9c5ac0-342e-4fca-838f-f2871af96e55-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854186 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9c5ac0-342e-4fca-838f-f2871af96e55-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.854451 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.854195 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrvx\" (UniqueName: \"kubernetes.io/projected/4e9c5ac0-342e-4fca-838f-f2871af96e55-kube-api-access-mdrvx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.878768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.878675 4920 generic.go:356] "Generic (PLEG): container finished" podID="3528fc4f-2bf6-499b-965a-a1738c881609" containerID="c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6" exitCode=0 Oct 30 09:59:23.878768 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.878736 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" Oct 30 09:59:23.878987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.878758 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" event={"ID":"3528fc4f-2bf6-499b-965a-a1738c881609","Type":"ContainerDied","Data":"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6"} Oct 30 09:59:23.878987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.878814 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v" event={"ID":"3528fc4f-2bf6-499b-965a-a1738c881609","Type":"ContainerDied","Data":"dc35a2fac055db559a67fa08173976d9463e4b7d8407ea55c91d7107211d00a2"} Oct 30 09:59:23.878987 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.878837 4920 scope.go:122] "RemoveContainer" containerID="c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6" Oct 30 09:59:23.880810 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.880790 4920 generic.go:356] "Generic (PLEG): container finished" podID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerID="29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102" exitCode=0 Oct 30 09:59:23.880925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.880841 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerDied","Data":"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102"} Oct 30 09:59:23.880925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.880896 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" event={"ID":"4e9c5ac0-342e-4fca-838f-f2871af96e55","Type":"ContainerDied","Data":"cda26b901cdbfa068e8c1818d5f8ca6bcf791cc5574d418b4025788d488f2d69"} Oct 30 09:59:23.880925 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.880849 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg" Oct 30 09:59:23.891105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.891084 4920 scope.go:122] "RemoveContainer" containerID="c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6" Oct 30 09:59:23.891464 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:23.891438 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6\": container with ID starting with c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6 not found: ID does not exist" containerID="c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6" Oct 30 09:59:23.891522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.891479 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6"} err="failed to get container status \"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6\": rpc error: code = NotFound desc = could not find container \"c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6\": container with ID starting with c1c9992857d0e1dfbaa0005d344b071bdecc653b57b1e369b80edebc582078c6 not found: ID does not exist" Oct 30 09:59:23.891522 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.891505 4920 scope.go:122] "RemoveContainer" containerID="29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102" Oct 30 09:59:23.901549 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.901523 4920 scope.go:122] "RemoveContainer" containerID="eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956" Oct 30 09:59:23.911800 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.911769 4920 scope.go:122] "RemoveContainer" containerID="29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102" Oct 30 09:59:23.912141 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:23.912119 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102\": container with ID starting with 29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102 not found: ID does not exist" containerID="29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102" Oct 30 09:59:23.912238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.912148 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102"} err="failed to get container status \"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102\": rpc error: code = NotFound desc = could not find container \"29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102\": container with ID starting with 29086c133a327a34d42fab46bff39c3b91f669e4cc7d55dc28a7ada5b4191102 not found: ID does not exist" Oct 30 09:59:23.912238 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.912170 4920 scope.go:122] "RemoveContainer" containerID="eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956" Oct 30 09:59:23.912473 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:23.912456 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956\": container with ID starting with eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956 not found: ID does not exist" containerID="eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956" Oct 30 09:59:23.912541 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.912478 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956"} err="failed to get container status \"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956\": rpc error: code = NotFound desc = could not find container \"eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956\": container with ID starting with eebd3265d13ce6b961989a51135cab4e6d6ea3150aff2ae07d87348fc26ed956 not found: ID does not exist" Oct 30 09:59:23.938724 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.938677 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:59:23.942033 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.942002 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6626v"] Oct 30 09:59:23.976013 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.975978 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:59:23.979061 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:23.979036 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-z8fkg"] Oct 30 09:59:24.397911 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:24.397866 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" path="/var/lib/kubelet/pods/3528fc4f-2bf6-499b-965a-a1738c881609/volumes" Oct 30 09:59:24.398425 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:24.398373 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" path="/var/lib/kubelet/pods/4e9c5ac0-342e-4fca-838f-f2871af96e55/volumes" Oct 30 09:59:24.548658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:24.548622 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:25.499998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:25.499959 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:59:26.115358 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:26.115323 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:27.539267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.532436 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:59:27.539267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.532645 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" containerID="cri-o://1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6" gracePeriod=30 Oct 30 09:59:27.547190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.544945 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:59:27.547190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.545269 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" containerID="cri-o://2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93" gracePeriod=30 Oct 30 09:59:27.563658 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.563612 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:59:27.564191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.564154 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" containerID="cri-o://29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10" gracePeriod=30 Oct 30 09:59:27.572203 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.572173 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:27.968334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:27.968250 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:28.074832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:28.074794 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:28.295267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:28.288526 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:59:28.295267 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:28.288802 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" containerID="cri-o://73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f" gracePeriod=30 Oct 30 09:59:28.317472 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:28.317436 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:28.822755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:28.822663 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:29.258887 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:29.258801 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:29.559261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:29.559169 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:29.666181 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:29.666133 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:30.861054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:30.861021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:31.137559 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:31.137470 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 09:59:32.464951 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.464919 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:32.877571 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.877538 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:59:32.888370 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888336 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888402 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888516 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888435 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888608 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888515 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888664 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888632 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.888740 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888710 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888756 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888799 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888787 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888822 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.888914 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.888858 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\") pod \"00c7162e-8cce-4604-a97e-057171973a45\" (UID: \"00c7162e-8cce-4604-a97e-057171973a45\") " Oct 30 09:59:32.889331 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.889306 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.889576 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.889552 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.889990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.889769 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.889990 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.889959 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.890139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.890060 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:32.892533 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.892506 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.894790 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.894659 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.896427 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.896390 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:32.899429 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.899391 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc" pod "00c7162e-8cce-4604-a97e-057171973a45" (UID: "00c7162e-8cce-4604-a97e-057171973a45"). InnerVolumeSpecName "kube-api-access-cvptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.906105 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.906077 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:59:32.908447 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.908426 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:59:32.975908 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.975871 4920 generic.go:356] "Generic (PLEG): container finished" podID="00c7162e-8cce-4604-a97e-057171973a45" containerID="1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6" exitCode=0 Oct 30 09:59:32.976078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.975920 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" Oct 30 09:59:32.976078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.975959 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" event={"ID":"00c7162e-8cce-4604-a97e-057171973a45","Type":"ContainerDied","Data":"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6"} Oct 30 09:59:32.976078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.976048 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w" event={"ID":"00c7162e-8cce-4604-a97e-057171973a45","Type":"ContainerDied","Data":"322b91e7606533b34f04cd84af029bcabd0cb21e847a3dd9190af54046c058e1"} Oct 30 09:59:32.976078 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.976076 4920 scope.go:122] "RemoveContainer" containerID="1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6" Oct 30 09:59:32.977745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.977724 4920 generic.go:356] "Generic (PLEG): container finished" podID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerID="29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10" exitCode=0 Oct 30 09:59:32.977871 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.977811 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" Oct 30 09:59:32.977944 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.977808 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" event={"ID":"ab3ca2c6-df41-474b-981e-388470fc0bea","Type":"ContainerDied","Data":"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10"} Oct 30 09:59:32.978004 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.977970 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm" event={"ID":"ab3ca2c6-df41-474b-981e-388470fc0bea","Type":"ContainerDied","Data":"fd6bca17c2eaf843cc9270495aeeea99bd1d7b5ca8ff706ce65e4da37d804d48"} Oct 30 09:59:32.979386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.979365 4920 generic.go:356] "Generic (PLEG): container finished" podID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerID="2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93" exitCode=0 Oct 30 09:59:32.979487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.979410 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" Oct 30 09:59:32.979487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.979420 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" event={"ID":"23a44536-43a9-4d18-986a-baa4cbff5cff","Type":"ContainerDied","Data":"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93"} Oct 30 09:59:32.979487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.979458 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk" event={"ID":"23a44536-43a9-4d18-986a-baa4cbff5cff","Type":"ContainerDied","Data":"e0a069a43e26011187922cbb3c7b86f0b7d49f8e04e9dd1ca5422f5291c85ba5"} Oct 30 09:59:32.989696 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989666 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.989822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989747 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.989822 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989796 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.989932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989834 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.989932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989860 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.990029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989929 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.990029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989971 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.990029 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.989994 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.990174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990025 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.990174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990070 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.990174 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990130 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990206 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990275 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990346 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990365 4920 scope.go:122] "RemoveContainer" containerID="1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990373 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990475 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990516 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\") pod \"ab3ca2c6-df41-474b-981e-388470fc0bea\" (UID: \"ab3ca2c6-df41-474b-981e-388470fc0bea\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990566 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\") pod \"23a44536-43a9-4d18-986a-baa4cbff5cff\" (UID: \"23a44536-43a9-4d18-986a-baa4cbff5cff\") " Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990617 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990829 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990974 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7162e-8cce-4604-a97e-057171973a45-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990988 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvptc\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-kube-api-access-cvptc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.990997 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991007 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991015 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991023 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7162e-8cce-4604-a97e-057171973a45-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991031 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7162e-8cce-4604-a97e-057171973a45-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991039 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991047 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7162e-8cce-4604-a97e-057171973a45-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991055 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991415 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991491 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.991977 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.991822 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.992075 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.992393 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:32.993062 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6\": container with ID starting with 1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6 not found: ID does not exist" containerID="1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.993102 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6"} err="failed to get container status \"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6\": rpc error: code = NotFound desc = could not find container \"1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6\": container with ID starting with 1f3f27f7d7d5b41c39780a0e87d97dfc8b4f6958a7c561c621db2fcf799c08c6 not found: ID does not exist" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.993128 4920 scope.go:122] "RemoveContainer" containerID="29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.993165 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.993487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.993443 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.993882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.993840 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:32.994139 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.994112 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.994718 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.994464 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "kube-api-access-f2s6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.994829 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.994721 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.995550 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.995423 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.996185 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.996161 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:32.996390 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.996367 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "kube-api-access-gg5mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.996463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:32.996369 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token" pod "ab3ca2c6-df41-474b-981e-388470fc0bea" (UID: "ab3ca2c6-df41-474b-981e-388470fc0bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:33.007580 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.007555 4920 scope.go:122] "RemoveContainer" containerID="29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10" Oct 30 09:59:33.007982 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:33.007948 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10\": container with ID starting with 29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10 not found: ID does not exist" containerID="29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10" Oct 30 09:59:33.008130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.007990 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10"} err="failed to get container status \"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10\": rpc error: code = NotFound desc = could not find container \"29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10\": container with ID starting with 29ea1fd699699d4deca620f3527b7a48dcdf3733deb312ba7d74aed8205d6d10 not found: ID does not exist" Oct 30 09:59:33.008130 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.008014 4920 scope.go:122] "RemoveContainer" containerID="2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93" Oct 30 09:59:33.014762 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.014722 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo" pod "23a44536-43a9-4d18-986a-baa4cbff5cff" (UID: "23a44536-43a9-4d18-986a-baa4cbff5cff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:33.032826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.032793 4920 scope.go:122] "RemoveContainer" containerID="2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93" Oct 30 09:59:33.033307 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:33.033276 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93\": container with ID starting with 2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93 not found: ID does not exist" containerID="2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93" Oct 30 09:59:33.033499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.033470 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93"} err="failed to get container status \"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93\": rpc error: code = NotFound desc = could not find container \"2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93\": container with ID starting with 2462b57f849853397e0b2a5f4c16e776c34c5d9b66d1f2439f8e2fb80f9fdb93 not found: ID does not exist" Oct 30 09:59:33.077610 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.077572 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:59:33.080302 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.080277 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lpk2w"] Oct 30 09:59:33.091965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091928 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.091965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091961 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.091965 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091970 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091977 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091986 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.091994 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092002 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092010 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2s6s\" (UniqueName: \"kubernetes.io/projected/23a44536-43a9-4d18-986a-baa4cbff5cff-kube-api-access-f2s6s\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092018 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092027 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092037 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5mz\" (UniqueName: \"kubernetes.io/projected/ab3ca2c6-df41-474b-981e-388470fc0bea-kube-api-access-gg5mz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092045 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092053 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab3ca2c6-df41-474b-981e-388470fc0bea-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092062 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab3ca2c6-df41-474b-981e-388470fc0bea-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092070 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a44536-43a9-4d18-986a-baa4cbff5cff-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.092190 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.092077 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a44536-43a9-4d18-986a-baa4cbff5cff-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.326191 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.326157 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:59:33.328941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.328912 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tjlgk"] Oct 30 09:59:33.359933 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.359896 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:59:33.363261 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.363236 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-knmrm"] Oct 30 09:59:33.548345 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.548318 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:59:33.596844 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.596812 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.596999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.596939 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.596999 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.596964 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597006 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597050 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597087 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597129 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597112 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597170 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597230 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\") pod \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\" (UID: \"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3\") " Oct 30 09:59:33.597332 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597238 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:33.597496 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597462 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:33.597744 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597722 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:33.597826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597781 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.597826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.597803 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.598337 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.598286 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:33.601110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.600765 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:33.601110 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.600804 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "kube-api-access-zh7nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:33.601338 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.601167 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:33.602786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.602757 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:33.606243 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.606190 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo" pod "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" (UID: "9fb95ad4-6def-4bba-ba52-d7ba00cd8df3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:33.699152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699112 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699152 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699152 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699168 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7nh\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-kube-api-access-zh7nh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699183 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699197 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699209 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.699395 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.699252 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:33.790085 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.789993 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:59:33.873258 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.873201 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:33.967782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.967748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:33.989998 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.989967 4920 generic.go:356] "Generic (PLEG): container finished" podID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerID="73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f" exitCode=0 Oct 30 09:59:33.990183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.990017 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" Oct 30 09:59:33.990183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.990035 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerDied","Data":"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f"} Oct 30 09:59:33.990183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.990084 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj" event={"ID":"9fb95ad4-6def-4bba-ba52-d7ba00cd8df3","Type":"ContainerDied","Data":"83d677466bcbe8088dbc50654f3504b2cb44cebac00d1578574020eead6b0e86"} Oct 30 09:59:33.990183 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:33.990114 4920 scope.go:122] "RemoveContainer" containerID="73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f" Oct 30 09:59:34.000772 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.000754 4920 scope.go:122] "RemoveContainer" containerID="2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587" Oct 30 09:59:34.010157 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.010139 4920 scope.go:122] "RemoveContainer" containerID="73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f" Oct 30 09:59:34.010467 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:34.010447 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f\": container with ID starting with 73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f not found: ID does not exist" containerID="73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f" Oct 30 09:59:34.010540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.010477 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f"} err="failed to get container status \"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f\": rpc error: code = NotFound desc = could not find container \"73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f\": container with ID starting with 73909b419d47e31f348cbfd0f5a28ec3142510743f2f327a2e35a4ee55cb6d3f not found: ID does not exist" Oct 30 09:59:34.010540 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.010494 4920 scope.go:122] "RemoveContainer" containerID="2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587" Oct 30 09:59:34.010790 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:34.010771 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587\": container with ID starting with 2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587 not found: ID does not exist" containerID="2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587" Oct 30 09:59:34.010846 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.010795 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587"} err="failed to get container status \"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587\": rpc error: code = NotFound desc = could not find container \"2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587\": container with ID starting with 2386ce9155906365015a2f7b713ca9ab2aa95637fe957a8e375c2657185e7587 not found: ID does not exist" Oct 30 09:59:34.050434 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.050341 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:59:34.053209 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.053180 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c88ae-openshift-default-5b4fb76df7-nbjcj"] Oct 30 09:59:34.399441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.399347 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c7162e-8cce-4604-a97e-057171973a45" path="/var/lib/kubelet/pods/00c7162e-8cce-4604-a97e-057171973a45/volumes" Oct 30 09:59:34.400027 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.400004 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" path="/var/lib/kubelet/pods/23a44536-43a9-4d18-986a-baa4cbff5cff/volumes" Oct 30 09:59:34.400672 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.400649 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" path="/var/lib/kubelet/pods/9fb95ad4-6def-4bba-ba52-d7ba00cd8df3/volumes" Oct 30 09:59:34.401792 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:34.401765 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" path="/var/lib/kubelet/pods/ab3ca2c6-df41-474b-981e-388470fc0bea/volumes" Oct 30 09:59:35.525097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:35.525053 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:37.024984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:37.024944 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:38.623572 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:38.623537 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:39.698051 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:39.698001 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:39.806581 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:39.804788 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:39.843314 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:39.843270 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:59:39.843528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:39.843501 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" podUID="f2eebedf-163f-4cd0-9c4f-61483f191427" containerName="maas-api" containerID="cri-o://815cbf4fb0959f9e2dba81b93dfb60d4b6006f28fdd7fc210b18f089897219c2" gracePeriod=30 Oct 30 09:59:39.900131 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:39.900092 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:40.041542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.039047 4920 generic.go:356] "Generic (PLEG): container finished" podID="f2eebedf-163f-4cd0-9c4f-61483f191427" containerID="815cbf4fb0959f9e2dba81b93dfb60d4b6006f28fdd7fc210b18f089897219c2" exitCode=0 Oct 30 09:59:40.041542 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.039116 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" event={"ID":"f2eebedf-163f-4cd0-9c4f-61483f191427","Type":"ContainerDied","Data":"815cbf4fb0959f9e2dba81b93dfb60d4b6006f28fdd7fc210b18f089897219c2"} Oct 30 09:59:40.128605 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.128581 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:59:40.174385 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.174350 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\" (UniqueName: \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\") pod \"f2eebedf-163f-4cd0-9c4f-61483f191427\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " Oct 30 09:59:40.174569 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.174459 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\") pod \"f2eebedf-163f-4cd0-9c4f-61483f191427\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " Oct 30 09:59:40.174613 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.174581 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") pod \"f2eebedf-163f-4cd0-9c4f-61483f191427\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " Oct 30 09:59:40.174655 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.174632 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\") pod \"f2eebedf-163f-4cd0-9c4f-61483f191427\" (UID: \"f2eebedf-163f-4cd0-9c4f-61483f191427\") " Oct 30 09:59:40.175162 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.175143 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca" pod "f2eebedf-163f-4cd0-9c4f-61483f191427" (UID: "f2eebedf-163f-4cd0-9c4f-61483f191427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:40.177433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.177399 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs" pod "f2eebedf-163f-4cd0-9c4f-61483f191427" (UID: "f2eebedf-163f-4cd0-9c4f-61483f191427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:40.177433 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.177425 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp" pod "f2eebedf-163f-4cd0-9c4f-61483f191427" (UID: "f2eebedf-163f-4cd0-9c4f-61483f191427"). InnerVolumeSpecName "kube-api-access-rvjjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:40.177621 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.177486 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls" pod "f2eebedf-163f-4cd0-9c4f-61483f191427" (UID: "f2eebedf-163f-4cd0-9c4f-61483f191427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:40.276106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.276020 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:40.276106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.276062 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2eebedf-163f-4cd0-9c4f-61483f191427-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:40.276106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.276077 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2eebedf-163f-4cd0-9c4f-61483f191427-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:40.276106 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.276092 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjjp\" (UniqueName: \"kubernetes.io/projected/f2eebedf-163f-4cd0-9c4f-61483f191427-kube-api-access-rvjjp\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:40.351960 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.351912 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:40.452280 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.452245 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:40.553480 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.553448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:40.991454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:40.991371 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:41.047123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:41.047092 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" Oct 30 09:59:41.047123 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:41.047102 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw" event={"ID":"f2eebedf-163f-4cd0-9c4f-61483f191427","Type":"ContainerDied","Data":"271c23a9311851fb48d10a3d5bdcdb2981941f6ad4ab22a88a78b6ca2d9ca77f"} Oct 30 09:59:41.047416 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:41.047170 4920 scope.go:122] "RemoveContainer" containerID="815cbf4fb0959f9e2dba81b93dfb60d4b6006f28fdd7fc210b18f089897219c2" Oct 30 09:59:41.097525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:41.097485 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:59:41.110454 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:41.110420 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a479df-6ff556d977-t2nlw"] Oct 30 09:59:42.084545 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:42.084515 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:42.398127 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:42.398049 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2eebedf-163f-4cd0-9c4f-61483f191427" path="/var/lib/kubelet/pods/f2eebedf-163f-4cd0-9c4f-61483f191427/volumes" Oct 30 09:59:43.551334 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:43.551303 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:45.050326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:45.050283 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:45.091728 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:45.091692 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 09:59:46.647485 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:46.647451 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:46.896463 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:46.896430 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:46.997657 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:46.997562 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:47.235252 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:47.235196 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:59:47.235528 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:47.235499 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" containerID="cri-o://a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad" gracePeriod=30 Oct 30 09:59:47.255525 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:47.255435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:48.266885 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:48.266851 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:48.816441 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:48.816392 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" containerID="cri-o://fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca" gracePeriod=45 Oct 30 09:59:49.075676 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.075616 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:59:49.103112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.103073 4920 generic.go:356] "Generic (PLEG): container finished" podID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerID="fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca" exitCode=0 Oct 30 09:59:49.103112 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.103105 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" event={"ID":"90ecec18-5a58-4ed0-8831-aa0f3a23d51f","Type":"ContainerDied","Data":"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca"} Oct 30 09:59:49.103378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.103125 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" Oct 30 09:59:49.103378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.103167 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm" event={"ID":"90ecec18-5a58-4ed0-8831-aa0f3a23d51f","Type":"ContainerDied","Data":"1e5594f9e258b4afab8eb14d6a71f4868edf8bd4096e85d86770c7de5d1e1dfb"} Oct 30 09:59:49.103378 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.103191 4920 scope.go:122] "RemoveContainer" containerID="fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca" Oct 30 09:59:49.114673 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.114639 4920 scope.go:122] "RemoveContainer" containerID="fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca" Oct 30 09:59:49.115009 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:49.114984 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca\": container with ID starting with fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca not found: ID does not exist" containerID="fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca" Oct 30 09:59:49.115097 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.115019 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca"} err="failed to get container status \"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca\": rpc error: code = NotFound desc = could not find container \"fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca\": container with ID starting with fad04597b8df8719c19c7e7e21ad70824cbc723514e6dff1501e4cf02d7d23ca not found: ID does not exist" Oct 30 09:59:49.202306 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202272 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202318 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202352 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\" (UniqueName: \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202408 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202510 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202471 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\" (UniqueName: \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202576 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\") pod \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\" (UID: \"90ecec18-5a58-4ed0-8831-aa0f3a23d51f\") " Oct 30 09:59:49.202755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202593 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.202932 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.202806 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.203196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.203136 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.203196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.203162 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.203436 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.203292 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.205769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.205443 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:49.205769 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.205566 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:49.206172 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.206148 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd" pod "90ecec18-5a58-4ed0-8831-aa0f3a23d51f" (UID: "90ecec18-5a58-4ed0-8831-aa0f3a23d51f"). InnerVolumeSpecName "kube-api-access-vx7rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:49.305196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.305163 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7rd\" (UniqueName: \"kubernetes.io/projected/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-kube-api-access-vx7rd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.305196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.305193 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.305196 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.305205 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.305813 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.305247 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90ecec18-5a58-4ed0-8831-aa0f3a23d51f-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:49.448907 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.448865 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:59:49.451382 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.451351 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3a61e1/distinct-2-w4-3a61e1-kserve-7d8b88596f-fckbm"] Oct 30 09:59:49.555934 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.555900 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b"] Oct 30 09:59:49.556499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556478 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" Oct 30 09:59:49.556499 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556499 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556517 4920 cpu_manager.go:414] "containerMap: removing container" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556525 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556539 4920 cpu_manager.go:414] "containerMap: removing container" podUID="1b6595eb-f73c-4c26-89c9-3a693410528f" containerName="maas-api" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556546 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6595eb-f73c-4c26-89c9-3a693410528f" containerName="maas-api" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556559 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556567 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556580 4920 cpu_manager.go:414] "containerMap: removing container" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556588 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556598 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556606 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556622 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556629 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556638 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556645 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556654 4920 cpu_manager.go:414] "containerMap: removing container" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556664 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556674 4920 cpu_manager.go:414] "containerMap: removing container" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556682 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556693 4920 cpu_manager.go:414] "containerMap: removing container" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556700 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" Oct 30 09:59:49.556703 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556712 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ebf964b5-9081-43a1-8345-f60fb1a375eb" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556719 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf964b5-9081-43a1-8345-f60fb1a375eb" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556731 4920 cpu_manager.go:414] "containerMap: removing container" podUID="000155aa-ce40-4e39-a5b6-6961b01e88d1" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556739 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000155aa-ce40-4e39-a5b6-6961b01e88d1" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556750 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7b9de14d-2ed8-4644-806f-f034ed71b659" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556758 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9de14d-2ed8-4644-806f-f034ed71b659" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556770 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556779 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556796 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556804 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556814 4920 cpu_manager.go:414] "containerMap: removing container" podUID="094edb06-2aff-4715-8a39-86be7ed5746f" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556821 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094edb06-2aff-4715-8a39-86be7ed5746f" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556830 4920 cpu_manager.go:414] "containerMap: removing container" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556837 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556847 4920 cpu_manager.go:414] "containerMap: removing container" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556855 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556864 4920 cpu_manager.go:414] "containerMap: removing container" podUID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556871 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556882 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f2eebedf-163f-4cd0-9c4f-61483f191427" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556889 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2eebedf-163f-4cd0-9c4f-61483f191427" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556906 4920 cpu_manager.go:414] "containerMap: removing container" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556913 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556922 4920 cpu_manager.go:414] "containerMap: removing container" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556929 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556939 4920 cpu_manager.go:414] "containerMap: removing container" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556946 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556959 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556966 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556978 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556986 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.556997 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557004 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557014 4920 cpu_manager.go:414] "containerMap: removing container" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557021 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557030 4920 cpu_manager.go:414] "containerMap: removing container" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557037 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557059 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557067 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557079 4920 cpu_manager.go:414] "containerMap: removing container" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557086 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557097 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557106 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557119 4920 cpu_manager.go:414] "containerMap: removing container" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557127 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557138 4920 cpu_manager.go:414] "containerMap: removing container" podUID="9b5212c0-889d-445c-ad41-7abf90a81555" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557144 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5212c0-889d-445c-ad41-7abf90a81555" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557154 4920 cpu_manager.go:414] "containerMap: removing container" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557162 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557177 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557185 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557198 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557205 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557231 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557240 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557249 4920 cpu_manager.go:414] "containerMap: removing container" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557256 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557265 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557272 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557286 4920 cpu_manager.go:414] "containerMap: removing container" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557293 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557307 4920 cpu_manager.go:414] "containerMap: removing container" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557314 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557324 4920 cpu_manager.go:414] "containerMap: removing container" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557331 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557346 4920 cpu_manager.go:414] "containerMap: removing container" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557353 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557487 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b583ec7-6a62-4e11-9dfd-a6f4c2c06e37" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557503 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557514 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557523 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557534 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3528fc4f-2bf6-499b-965a-a1738c881609" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557543 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="32beebe9-cdc6-464b-af7e-50bce18f65f9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557552 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f56fec-70e4-4d0c-80e6-4bf5f4b3f32f" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557563 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557574 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5212c0-889d-445c-ad41-7abf90a81555" containerName="payload-pre-processing" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557583 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b96941-be95-4b6c-bd41-8d8cef2f4fd6" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557591 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557600 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557609 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557617 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5716962-c02e-4388-83b0-b11c9023bc90" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557624 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b41fc0-f9a2-4cb3-bd10-74700220b99c" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557634 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac433f30-005f-4f0e-94f3-779fb966312d" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557644 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca2f98d-2b90-4d40-82bf-abb15f0f7105" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557652 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ce10d9-d408-497c-9b17-42ec3433d78a" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557662 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a86afa-224a-40ea-8eba-14c2513182b1" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557672 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccea84a3-36c1-4d20-a018-59db9ed75f16" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557682 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557690 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557699 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdc64f0-df08-435a-8583-81fed012ba0f" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557709 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557717 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6595eb-f73c-4c26-89c9-3a693410528f" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557726 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c7162e-8cce-4604-a97e-057171973a45" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557736 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557745 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38f1e38-8109-41b7-84a1-8b83fa173850" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557754 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3ca2c6-df41-474b-981e-388470fc0bea" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557764 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4878df82-03f8-4d95-8173-9cb84d7a8c8c" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557772 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557781 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557790 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a44536-43a9-4d18-986a-baa4cbff5cff" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557798 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf964b5-9081-43a1-8345-f60fb1a375eb" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557807 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="000155aa-ce40-4e39-a5b6-6961b01e88d1" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557815 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9de14d-2ed8-4644-806f-f034ed71b659" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557823 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7795d4-23df-4ff0-8d58-89858fec7b26" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557832 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557839 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="094edb06-2aff-4715-8a39-86be7ed5746f" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557846 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dd6fe5-7347-46b0-8b0f-5c4f2b7c83b2" containerName="istio-proxy" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557854 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="6937d716-3d86-492b-b48a-42e96fbe40d4" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557862 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c4c19f-4acd-4b1d-9212-58651e8d4a97" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557872 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2eebedf-163f-4cd0-9c4f-61483f191427" containerName="maas-api" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557882 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="297160b5-6012-4f00-81c8-c01911769694" containerName="main" Oct 30 09:59:49.557832 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.557891 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9c5ac0-342e-4fca-838f-f2871af96e55" containerName="istio-proxy" Oct 30 09:59:49.563284 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.563263 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.577386 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.577359 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:59:49.577523 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.577454 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:59:49.578941 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.578913 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b"] Oct 30 09:59:49.711353 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.711262 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.711543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.711441 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.711543 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.711493 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngnk\" (UniqueName: \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.812774 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.812735 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.812979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.812840 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.812979 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.812882 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngnk\" (UniqueName: \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.813521 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.813499 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.815114 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.815093 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.834653 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.834623 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngnk\" (UniqueName: \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\") pod \"maas-api-revoke-sub-af9ebb7f-tws9b\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.867567 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.867538 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:49.879498 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.879465 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:49.921376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.921015 4920 patch_prober.go:29] interesting pod/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:59:49.921376 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:49.921075 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:59:50.197436 ip-10-0-129-105 kubenswrapper[4920]: W1030 09:59:50.197400 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61ebbd7_389b_4756_b6a0_ebcd1bb16f34.slice/crio-7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2 WatchSource:0}: Error finding container 7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2: Status 404 returned error can't find the container with id 7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2 Oct 30 09:59:50.198000 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:50.197971 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b"] Oct 30 09:59:50.399205 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:50.399171 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ecec18-5a58-4ed0-8831-aa0f3a23d51f" path="/var/lib/kubelet/pods/90ecec18-5a58-4ed0-8831-aa0f3a23d51f/volumes" Oct 30 09:59:51.118324 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:51.118294 4920 generic.go:356] "Generic (PLEG): container finished" podID="a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" containerID="9fbc011f9ca004b871054dc6918b2190dfecca79718978717aab5b5dd054698b" exitCode=0 Oct 30 09:59:51.118467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:51.118377 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" event={"ID":"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34","Type":"ContainerDied","Data":"9fbc011f9ca004b871054dc6918b2190dfecca79718978717aab5b5dd054698b"} Oct 30 09:59:51.118467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:51.118434 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" event={"ID":"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34","Type":"ContainerStarted","Data":"7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2"} Oct 30 09:59:51.466745 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:51.466619 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:52.367782 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.367754 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:52.442984 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.442947 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\") pod \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " Oct 30 09:59:52.443309 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.443286 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\" (UniqueName: \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\") pod \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " Oct 30 09:59:52.443457 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.443441 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\") pod \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\" (UID: \"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34\") " Oct 30 09:59:52.443935 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.443484 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca" pod "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" (UID: "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:52.446593 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.446572 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca" pod "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" (UID: "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:52.446737 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.446616 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk" pod "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" (UID: "a61ebbd7-389b-4756-b6a0-ebcd1bb16f34"). InnerVolumeSpecName "kube-api-access-pngnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:52.495467 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.495444 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:59:52.544638 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544591 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.544820 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544787 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.544883 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544832 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.544947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544872 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.544947 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544920 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.545062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544964 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.545062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.544995 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.545062 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545049 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.545255 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545077 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\") pod \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\" (UID: \"fc96b8ff-00f2-486f-9eaa-916e9695fdd7\") " Oct 30 09:59:52.545326 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545296 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:52.545500 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545478 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:52.545595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545553 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngnk\" (UniqueName: \"kubernetes.io/projected/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-kube-api-access-pngnk\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.545595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545574 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.545595 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545587 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.545763 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545600 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a61ebbd7-389b-4756-b6a0-ebcd1bb16f34-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.545830 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.545771 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:52.546149 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.546124 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:52.546487 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.546463 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:52.547860 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.547836 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:52.547949 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.547885 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:52.548136 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.548115 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:52.548336 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.548315 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc" pod "fc96b8ff-00f2-486f-9eaa-916e9695fdd7" (UID: "fc96b8ff-00f2-486f-9eaa-916e9695fdd7"). InnerVolumeSpecName "kube-api-access-4xzbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:52.647050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647010 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647045 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647050 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647058 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647070 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647081 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647094 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647106 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzbc\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-kube-api-access-4xzbc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.647355 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.647119 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96b8ff-00f2-486f-9eaa-916e9695fdd7-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.960789 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:52.960755 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:53.136282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.136250 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" Oct 30 09:59:53.136282 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.136263 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9ebb7f-tws9b" event={"ID":"a61ebbd7-389b-4756-b6a0-ebcd1bb16f34","Type":"ContainerDied","Data":"7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2"} Oct 30 09:59:53.136502 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.136320 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a17197f240470fd91a2e96fb93c975f03058163c057843f0b95f51dba9083b2" Oct 30 09:59:53.138268 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.138212 4920 generic.go:356] "Generic (PLEG): container finished" podID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerID="a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad" exitCode=0 Oct 30 09:59:53.138415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.138275 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerDied","Data":"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad"} Oct 30 09:59:53.138415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.138300 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" Oct 30 09:59:53.138415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.138314 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q" event={"ID":"fc96b8ff-00f2-486f-9eaa-916e9695fdd7","Type":"ContainerDied","Data":"269297352fd0af3cdcc4c93aa779a00b5ef314766e6d1da635f0a1a7daf9ac6d"} Oct 30 09:59:53.138415 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.138342 4920 scope.go:122] "RemoveContainer" containerID="a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad" Oct 30 09:59:53.148755 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.148733 4920 scope.go:122] "RemoveContainer" containerID="9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4" Oct 30 09:59:53.159054 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.159033 4920 scope.go:122] "RemoveContainer" containerID="a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad" Oct 30 09:59:53.159398 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:53.159376 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad\": container with ID starting with a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad not found: ID does not exist" containerID="a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad" Oct 30 09:59:53.159466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.159407 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad"} err="failed to get container status \"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad\": rpc error: code = NotFound desc = could not find container \"a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad\": container with ID starting with a3d22eb818022b3e236c2e62f805877597f1c43707a105be033fc631f3be92ad not found: ID does not exist" Oct 30 09:59:53.159466 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.159427 4920 scope.go:122] "RemoveContainer" containerID="9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4" Oct 30 09:59:53.159688 ip-10-0-129-105 kubenswrapper[4920]: E1030 09:59:53.159668 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4\": container with ID starting with 9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4 not found: ID does not exist" containerID="9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4" Oct 30 09:59:53.159751 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.159698 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4"} err="failed to get container status \"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4\": rpc error: code = NotFound desc = could not find container \"9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4\": container with ID starting with 9b3af87e77bec20c46931a556b7e7f7ea07e5e6e9272c4bfef8277be8f9b52a4 not found: ID does not exist" Oct 30 09:59:53.204826 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.204791 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:59:53.216651 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:53.216571 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a479df-openshift-default-66bdb8fd8c-qnd7q"] Oct 30 09:59:54.397660 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:54.397620 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" path="/var/lib/kubelet/pods/fc96b8ff-00f2-486f-9eaa-916e9695fdd7/volumes" Oct 30 09:59:54.559142 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:54.559096 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:54.662438 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:54.662351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:56.176065 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:56.176031 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:57.762030 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:57.761997 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:58.418909 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:58.418875 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:58.533882 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:58.533611 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 09:59:59.255786 ip-10-0-129-105 kubenswrapper[4920]: I1030 09:59:59.255742 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:00.141452 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.141415 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:00.142110 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142090 4920 cpu_manager.go:414] "containerMap: removing container" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 10:00:00.142167 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142114 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bd5027-e611-438c-a0cb-c3b092154d30" containerName="istio-proxy" Oct 30 10:00:00.142167 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142137 4920 cpu_manager.go:414] "containerMap: removing container" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 10:00:00.142167 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142146 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054f2f7e-0462-4f88-8cf1-56888d1d20d0" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142169 4920 cpu_manager.go:414] "containerMap: removing container" podUID="a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" containerName="revoke-keys" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142177 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" containerName="revoke-keys" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142194 4920 cpu_manager.go:414] "containerMap: removing container" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142201 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142232 4920 cpu_manager.go:414] "containerMap: removing container" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142240 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142274 4920 cpu_manager.go:414] "containerMap: removing container" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142282 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142308 4920 cpu_manager.go:414] "containerMap: removing container" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.142315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142315 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.142628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142427 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.142628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142440 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafe5514-75c3-4add-95bb-92b5759017a9" containerName="istio-proxy" Oct 30 10:00:00.142628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142456 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb95ad4-6def-4bba-ba52-d7ba00cd8df3" containerName="istio-proxy" Oct 30 10:00:00.142628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142465 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61ebbd7-389b-4756-b6a0-ebcd1bb16f34" containerName="revoke-keys" Oct 30 10:00:00.142628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.142478 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="765b68b6-9f92-4c9b-b295-dafb87c8ab4e" containerName="istio-proxy" Oct 30 10:00:00.147978 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.147957 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:00.148090 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.148074 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.148800 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.148776 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96b8ff-00f2-486f-9eaa-916e9695fdd7" containerName="istio-proxy" Oct 30 10:00:00.151814 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.151797 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.161887 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.161860 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:00.182142 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.181992 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:00.182142 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.182111 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.191619 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.191583 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:00.207645 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.207615 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:00.230269 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230238 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.230437 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230402 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjh6\" (UniqueName: \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.230513 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230451 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jht5m\" (UniqueName: \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.230513 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230500 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.230695 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230531 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zflv\" (UniqueName: \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.230695 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.230598 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.265612 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.265587 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:00:00.331707 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331662 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjh6\" (UniqueName: \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.331707 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331707 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jht5m\" (UniqueName: \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.331996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331731 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.331996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331751 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zflv\" (UniqueName: \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.331996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331809 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.331996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.331867 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.332632 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.332607 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.332717 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.332607 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.332717 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.332606 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.350475 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.350449 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jht5m\" (UniqueName: \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.350636 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.350476 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zflv\" (UniqueName: \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.350636 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.350609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjh6\" (UniqueName: \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.474315 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.474237 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:00.510285 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.510252 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:00.523449 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.523418 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:00.780293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.780262 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:00.781060 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:00:00.781034 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f90dfa0_6bbe_4348_ae10_17ad14fec97f.slice/crio-26b1bb8c4557cb5564ec545c575ad7805c4992539cae8373ad81118a5010f128 WatchSource:0}: Error finding container 26b1bb8c4557cb5564ec545c575ad7805c4992539cae8373ad81118a5010f128: Status 404 returned error can't find the container with id 26b1bb8c4557cb5564ec545c575ad7805c4992539cae8373ad81118a5010f128 Oct 30 10:00:00.781420 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.781398 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:00.851267 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.846423 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:00.854469 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:00.854431 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:00.859259 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:00:00.857777 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37683065_673f_4bf6_bc1b_6424ad8c7765.slice/crio-df208a33189982bb992799e9a248f624d1036d863aedaecd114bc0117987ef1d WatchSource:0}: Error finding container df208a33189982bb992799e9a248f624d1036d863aedaecd114bc0117987ef1d: Status 404 returned error can't find the container with id df208a33189982bb992799e9a248f624d1036d863aedaecd114bc0117987ef1d Oct 30 10:00:01.140398 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:01.140359 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:00:01.221296 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.221257 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerStarted","Data":"d7b8228ca0b524e6e9801fb8f578a27c59437a767bbf3d8861ac42c798e1e545"} Oct 30 10:00:01.221456 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.221325 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerStarted","Data":"df208a33189982bb992799e9a248f624d1036d863aedaecd114bc0117987ef1d"} Oct 30 10:00:01.222967 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.222937 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerStarted","Data":"81d96ca8316b02158dde39b3b3e02ff988f640cfdd986d9da63e48339053c4c5"} Oct 30 10:00:01.223502 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.223442 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerStarted","Data":"e29fa4de3fb6dc05d7db6e4293bf21165ffb4c9789eab7d16f723f86e4cace6a"} Oct 30 10:00:01.236874 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.236825 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerStarted","Data":"1d93ca2fff388a7ac541d37c254c0c1cbf37b855a6d3ea8e68af3d2b39a1fb74"} Oct 30 10:00:01.237082 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.236929 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerStarted","Data":"26b1bb8c4557cb5564ec545c575ad7805c4992539cae8373ad81118a5010f128"} Oct 30 10:00:01.262069 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.262018 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" podStartSLOduration=1.2620051110000001 podStartE2EDuration="1.262005111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.259703907 +0000 UTC m=+3964.983131051" watchObservedRunningTime="2026-10-30 10:00:01.262005111 +0000 UTC m=+3964.985432254" Oct 30 10:00:01.296039 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.295981 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" podStartSLOduration=1.2959669520000001 podStartE2EDuration="1.295966952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.295084004 +0000 UTC m=+3965.018511150" watchObservedRunningTime="2026-10-30 10:00:01.295966952 +0000 UTC m=+3965.019394094" Oct 30 10:00:01.332662 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:01.332598 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" podStartSLOduration=1.332578834 podStartE2EDuration="1.332578834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.331617327 +0000 UTC m=+3965.055044494" watchObservedRunningTime="2026-10-30 10:00:01.332578834 +0000 UTC m=+3965.056005977" Oct 30 10:00:02.245713 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.245674 4920 generic.go:356] "Generic (PLEG): container finished" podID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerID="d7b8228ca0b524e6e9801fb8f578a27c59437a767bbf3d8861ac42c798e1e545" exitCode=60 Oct 30 10:00:02.245904 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.245762 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerDied","Data":"d7b8228ca0b524e6e9801fb8f578a27c59437a767bbf3d8861ac42c798e1e545"} Oct 30 10:00:02.246030 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.246011 4920 scope.go:122] "RemoveContainer" containerID="d7b8228ca0b524e6e9801fb8f578a27c59437a767bbf3d8861ac42c798e1e545" Oct 30 10:00:02.247500 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.247471 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerID="81d96ca8316b02158dde39b3b3e02ff988f640cfdd986d9da63e48339053c4c5" exitCode=60 Oct 30 10:00:02.247650 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.247574 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerDied","Data":"81d96ca8316b02158dde39b3b3e02ff988f640cfdd986d9da63e48339053c4c5"} Oct 30 10:00:02.247861 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.247833 4920 scope.go:122] "RemoveContainer" containerID="81d96ca8316b02158dde39b3b3e02ff988f640cfdd986d9da63e48339053c4c5" Oct 30 10:00:02.249396 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.249261 4920 generic.go:356] "Generic (PLEG): container finished" podID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerID="1d93ca2fff388a7ac541d37c254c0c1cbf37b855a6d3ea8e68af3d2b39a1fb74" exitCode=60 Oct 30 10:00:02.249396 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.249344 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerDied","Data":"1d93ca2fff388a7ac541d37c254c0c1cbf37b855a6d3ea8e68af3d2b39a1fb74"} Oct 30 10:00:02.249541 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.249524 4920 scope.go:122] "RemoveContainer" containerID="1d93ca2fff388a7ac541d37c254c0c1cbf37b855a6d3ea8e68af3d2b39a1fb74" Oct 30 10:00:02.378263 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:02.378228 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:03.257554 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.257460 4920 generic.go:356] "Generic (PLEG): container finished" podID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerID="d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77" exitCode=60 Oct 30 10:00:03.257554 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.257538 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerDied","Data":"d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77"} Oct 30 10:00:03.257783 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.257597 4920 scope.go:122] "RemoveContainer" containerID="1d93ca2fff388a7ac541d37c254c0c1cbf37b855a6d3ea8e68af3d2b39a1fb74" Oct 30 10:00:03.257914 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.257895 4920 scope.go:122] "RemoveContainer" containerID="d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77" Oct 30 10:00:03.258210 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:03.258184 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk_odh-ai-gateway-infra(5f90dfa0-6bbe-4348-ae10-17ad14fec97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" Oct 30 10:00:03.259784 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.259762 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerStarted","Data":"67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6"} Oct 30 10:00:03.261568 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.261548 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerID="74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586" exitCode=60 Oct 30 10:00:03.261664 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.261579 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerDied","Data":"74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586"} Oct 30 10:00:03.261789 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.261775 4920 scope.go:122] "RemoveContainer" containerID="74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586" Oct 30 10:00:03.261978 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:03.261961 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg_odh-ai-gateway-infra(e0ea94c2-b95f-4263-8fcb-56e1867ef1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" Oct 30 10:00:03.302412 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.302360 4920 scope.go:122] "RemoveContainer" containerID="81d96ca8316b02158dde39b3b3e02ff988f640cfdd986d9da63e48339053c4c5" Oct 30 10:00:03.627659 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:03.627625 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:04.077000 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.076957 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:04.174524 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.174490 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:04.269547 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.269515 4920 scope.go:122] "RemoveContainer" containerID="d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77" Oct 30 10:00:04.269842 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:04.269814 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk_odh-ai-gateway-infra(5f90dfa0-6bbe-4348-ae10-17ad14fec97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" Oct 30 10:00:04.271323 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.271299 4920 generic.go:356] "Generic (PLEG): container finished" podID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerID="67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6" exitCode=60 Oct 30 10:00:04.271457 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.271381 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerDied","Data":"67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6"} Oct 30 10:00:04.271457 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.271440 4920 scope.go:122] "RemoveContainer" containerID="d7b8228ca0b524e6e9801fb8f578a27c59437a767bbf3d8861ac42c798e1e545" Oct 30 10:00:04.271641 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.271626 4920 scope.go:122] "RemoveContainer" containerID="67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6" Oct 30 10:00:04.271937 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:04.271912 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt_odh-ai-gateway-infra(37683065-673f-4bf6-bc1b-6424ad8c7765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" Oct 30 10:00:04.273427 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.273411 4920 scope.go:122] "RemoveContainer" containerID="74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586" Oct 30 10:00:04.273615 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:04.273600 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg_odh-ai-gateway-infra(e0ea94c2-b95f-4263-8fcb-56e1867ef1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" Oct 30 10:00:04.278861 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.278836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:04.731637 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:04.731606 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:05.282138 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:05.282107 4920 scope.go:122] "RemoveContainer" containerID="67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6" Oct 30 10:00:05.282548 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:05.282523 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt_odh-ai-gateway-infra(37683065-673f-4bf6-bc1b-6424ad8c7765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" Oct 30 10:00:05.722502 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:05.722471 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:06.173545 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:06.173513 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:06.276267 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:06.276200 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:06.361052 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:06.361021 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:06.804518 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:06.804480 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:07.860854 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:07.860820 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:08.240745 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:08.240650 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:00:09.470598 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:09.470560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:10.271420 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:10.271382 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:10.571201 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:10.571172 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:10.782560 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:10.782529 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:10.998141 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:10.998058 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:11.098543 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:11.098512 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:12.537751 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:12.537711 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:12.672319 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:12.672253 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:12.772234 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:12.772184 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:12.874527 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:12.874448 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:13.194538 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:13.194450 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:13.315347 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:13.315310 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:13.417065 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:13.417033 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:13.654766 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:13.654734 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:14.302557 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:14.302521 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:15.963105 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:15.963065 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:17.390847 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:17.390817 4920 scope.go:122] "RemoveContainer" containerID="67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6" Oct 30 10:00:17.391361 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:17.390982 4920 scope.go:122] "RemoveContainer" containerID="74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586" Oct 30 10:00:17.466822 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:17.466786 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:18.375364 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.375331 4920 generic.go:356] "Generic (PLEG): container finished" podID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerID="3ca8413bb3511e16367cb361f0756cbd36667cf202925a5fd403a4f66231edb3" exitCode=60 Oct 30 10:00:18.375567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.375405 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerDied","Data":"3ca8413bb3511e16367cb361f0756cbd36667cf202925a5fd403a4f66231edb3"} Oct 30 10:00:18.375567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.375491 4920 scope.go:122] "RemoveContainer" containerID="67197cfd7a297d5de840c7d046fa9744ea1d6f0a4a4c55e4964e9ee05f3048a6" Oct 30 10:00:18.375906 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.375881 4920 scope.go:122] "RemoveContainer" containerID="3ca8413bb3511e16367cb361f0756cbd36667cf202925a5fd403a4f66231edb3" Oct 30 10:00:18.376186 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:18.376153 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt_odh-ai-gateway-infra(37683065-673f-4bf6-bc1b-6424ad8c7765)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" Oct 30 10:00:18.377386 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.377363 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerStarted","Data":"c24f8a53cb8c51e9a0eaccec452f7a702c6ab0cd950df45768b7343944671c55"} Oct 30 10:00:18.397886 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:18.397855 4920 scope.go:122] "RemoveContainer" containerID="d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77" Oct 30 10:00:19.025925 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.025836 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:19.125868 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.125824 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:19.386858 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.386817 4920 generic.go:356] "Generic (PLEG): container finished" podID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerID="c24f8a53cb8c51e9a0eaccec452f7a702c6ab0cd950df45768b7343944671c55" exitCode=60 Oct 30 10:00:19.387074 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.386884 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerDied","Data":"c24f8a53cb8c51e9a0eaccec452f7a702c6ab0cd950df45768b7343944671c55"} Oct 30 10:00:19.387074 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.386950 4920 scope.go:122] "RemoveContainer" containerID="74d5d1582e594999501ec3bed676b32eedce53d1a75d4cdd50923dcb5cc00586" Oct 30 10:00:19.387391 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.387364 4920 scope.go:122] "RemoveContainer" containerID="c24f8a53cb8c51e9a0eaccec452f7a702c6ab0cd950df45768b7343944671c55" Oct 30 10:00:19.387672 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:19.387644 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg_odh-ai-gateway-infra(e0ea94c2-b95f-4263-8fcb-56e1867ef1ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" Oct 30 10:00:19.389074 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.389052 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerStarted","Data":"82bf877f8e8c454f75af860689904538dc4c9525e8b1aae0e83966400db411cf"} Oct 30 10:00:19.421669 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.421640 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:19.476478 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.476442 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:19.593762 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.593735 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:19.672381 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.672297 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\" (UniqueName: \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\") pod \"37683065-673f-4bf6-bc1b-6424ad8c7765\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " Oct 30 10:00:19.672563 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.672387 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\") pod \"37683065-673f-4bf6-bc1b-6424ad8c7765\" (UID: \"37683065-673f-4bf6-bc1b-6424ad8c7765\") " Oct 30 10:00:19.672755 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.672724 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca" pod "37683065-673f-4bf6-bc1b-6424ad8c7765" (UID: "37683065-673f-4bf6-bc1b-6424ad8c7765"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:19.672897 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.672803 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37683065-673f-4bf6-bc1b-6424ad8c7765-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.674593 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.674567 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m" pod "37683065-673f-4bf6-bc1b-6424ad8c7765" (UID: "37683065-673f-4bf6-bc1b-6424ad8c7765"). InnerVolumeSpecName "kube-api-access-jht5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:19.774341 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:19.774307 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jht5m\" (UniqueName: \"kubernetes.io/projected/37683065-673f-4bf6-bc1b-6424ad8c7765-kube-api-access-jht5m\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:20.397745 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.397715 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" Oct 30 10:00:20.397943 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.397890 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt" event={"ID":"37683065-673f-4bf6-bc1b-6424ad8c7765","Type":"ContainerDied","Data":"df208a33189982bb992799e9a248f624d1036d863aedaecd114bc0117987ef1d"} Oct 30 10:00:20.398024 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.397981 4920 scope.go:122] "RemoveContainer" containerID="3ca8413bb3511e16367cb361f0756cbd36667cf202925a5fd403a4f66231edb3" Oct 30 10:00:20.407151 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.407111 4920 generic.go:356] "Generic (PLEG): container finished" podID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerID="82bf877f8e8c454f75af860689904538dc4c9525e8b1aae0e83966400db411cf" exitCode=60 Oct 30 10:00:20.407341 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.407312 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerDied","Data":"82bf877f8e8c454f75af860689904538dc4c9525e8b1aae0e83966400db411cf"} Oct 30 10:00:20.407827 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.407805 4920 scope.go:122] "RemoveContainer" containerID="82bf877f8e8c454f75af860689904538dc4c9525e8b1aae0e83966400db411cf" Oct 30 10:00:20.408137 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:20.408114 4920 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk_odh-ai-gateway-infra(5f90dfa0-6bbe-4348-ae10-17ad14fec97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" Oct 30 10:00:20.410503 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.410483 4920 scope.go:122] "RemoveContainer" containerID="d5cc77edc01cc785cdf508660eb1f230d2ffc3b307ca1fedbe3a22f458d24e77" Oct 30 10:00:20.450958 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.450927 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:20.453432 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.453409 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-388653-29889240-n2nqt"] Oct 30 10:00:20.512030 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.511991 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:20.564167 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.564136 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:20.683734 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.683645 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\" (UniqueName: \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\") pod \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " Oct 30 10:00:20.683734 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.683727 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\") pod \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\" (UID: \"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba\") " Oct 30 10:00:20.684120 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.684097 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca" pod "e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" (UID: "e0ea94c2-b95f-4263-8fcb-56e1867ef1ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:20.685908 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.685887 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv" pod "e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" (UID: "e0ea94c2-b95f-4263-8fcb-56e1867ef1ba"). InnerVolumeSpecName "kube-api-access-7zflv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:20.761998 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.761959 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:20.784890 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.784856 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zflv\" (UniqueName: \"kubernetes.io/projected/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-kube-api-access-7zflv\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:20.784890 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:20.784884 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:21.415588 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.415557 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" Oct 30 10:00:21.415588 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.415567 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg" event={"ID":"e0ea94c2-b95f-4263-8fcb-56e1867ef1ba","Type":"ContainerDied","Data":"e29fa4de3fb6dc05d7db6e4293bf21165ffb4c9789eab7d16f723f86e4cace6a"} Oct 30 10:00:21.415880 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.415632 4920 scope.go:122] "RemoveContainer" containerID="c24f8a53cb8c51e9a0eaccec452f7a702c6ab0cd950df45768b7343944671c55" Oct 30 10:00:21.465543 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.465505 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:21.469953 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.469925 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889240-f8pcg"] Oct 30 10:00:21.558498 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.558475 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:21.694006 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.693929 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\" (UniqueName: \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\") pod \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " Oct 30 10:00:21.694156 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.694079 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\") pod \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\" (UID: \"5f90dfa0-6bbe-4348-ae10-17ad14fec97f\") " Oct 30 10:00:21.694486 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.694458 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca" pod "5f90dfa0-6bbe-4348-ae10-17ad14fec97f" (UID: "5f90dfa0-6bbe-4348-ae10-17ad14fec97f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:21.696012 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.695988 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6" pod "5f90dfa0-6bbe-4348-ae10-17ad14fec97f" (UID: "5f90dfa0-6bbe-4348-ae10-17ad14fec97f"). InnerVolumeSpecName "kube-api-access-jrjh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:21.795596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.795557 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:21.795596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:21.795588 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjh6\" (UniqueName: \"kubernetes.io/projected/5f90dfa0-6bbe-4348-ae10-17ad14fec97f-kube-api-access-jrjh6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:22.265201 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.265160 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:22.398738 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.398698 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" path="/var/lib/kubelet/pods/37683065-673f-4bf6-bc1b-6424ad8c7765/volumes" Oct 30 10:00:22.399144 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.399129 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" path="/var/lib/kubelet/pods/e0ea94c2-b95f-4263-8fcb-56e1867ef1ba/volumes" Oct 30 10:00:22.426402 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.426373 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" Oct 30 10:00:22.426402 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.426392 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk" event={"ID":"5f90dfa0-6bbe-4348-ae10-17ad14fec97f","Type":"ContainerDied","Data":"26b1bb8c4557cb5564ec545c575ad7805c4992539cae8373ad81118a5010f128"} Oct 30 10:00:22.426641 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.426451 4920 scope.go:122] "RemoveContainer" containerID="82bf877f8e8c454f75af860689904538dc4c9525e8b1aae0e83966400db411cf" Oct 30 10:00:22.475387 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.475350 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:22.486881 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:22.486853 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889240-26wkk"] Oct 30 10:00:23.855329 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:23.855290 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:24.398271 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:24.398232 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" path="/var/lib/kubelet/pods/5f90dfa0-6bbe-4348-ae10-17ad14fec97f/volumes" Oct 30 10:00:24.775672 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:24.775590 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:24.881679 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:24.881550 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:25.462539 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:25.462468 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:26.954729 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:26.954687 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:28.453262 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:28.453178 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:29.744128 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:29.744090 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 10:00:29.744619 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:29.744373 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" podUID="15fc342a-b869-45c4-9b57-8fce99b62bf6" containerName="maas-api" containerID="cri-o://f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066" gracePeriod=30 Oct 30 10:00:29.874977 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:29.874941 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:30.044202 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.044170 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 10:00:30.204905 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.204862 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\" (UniqueName: \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\") pod \"15fc342a-b869-45c4-9b57-8fce99b62bf6\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " Oct 30 10:00:30.205151 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.204958 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") pod \"15fc342a-b869-45c4-9b57-8fce99b62bf6\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " Oct 30 10:00:30.205151 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.205049 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\") pod \"15fc342a-b869-45c4-9b57-8fce99b62bf6\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " Oct 30 10:00:30.205151 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.205084 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\") pod \"15fc342a-b869-45c4-9b57-8fce99b62bf6\" (UID: \"15fc342a-b869-45c4-9b57-8fce99b62bf6\") " Oct 30 10:00:30.205531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.205499 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca" pod "15fc342a-b869-45c4-9b57-8fce99b62bf6" (UID: "15fc342a-b869-45c4-9b57-8fce99b62bf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:30.207371 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.207344 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq" pod "15fc342a-b869-45c4-9b57-8fce99b62bf6" (UID: "15fc342a-b869-45c4-9b57-8fce99b62bf6"). InnerVolumeSpecName "kube-api-access-9g6tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:30.207589 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.207569 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs" pod "15fc342a-b869-45c4-9b57-8fce99b62bf6" (UID: "15fc342a-b869-45c4-9b57-8fce99b62bf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:30.207684 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.207663 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls" pod "15fc342a-b869-45c4-9b57-8fce99b62bf6" (UID: "15fc342a-b869-45c4-9b57-8fce99b62bf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:30.306009 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.305976 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6tq\" (UniqueName: \"kubernetes.io/projected/15fc342a-b869-45c4-9b57-8fce99b62bf6-kube-api-access-9g6tq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:30.306009 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.306006 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:30.306009 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.306015 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fc342a-b869-45c4-9b57-8fce99b62bf6-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:30.306348 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.306023 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fc342a-b869-45c4-9b57-8fce99b62bf6-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:30.318339 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.318300 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:30.419966 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.419932 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:30.487030 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.486989 4920 generic.go:356] "Generic (PLEG): container finished" podID="15fc342a-b869-45c4-9b57-8fce99b62bf6" containerID="f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066" exitCode=0 Oct 30 10:00:30.487248 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.487055 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" Oct 30 10:00:30.487248 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.487074 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" event={"ID":"15fc342a-b869-45c4-9b57-8fce99b62bf6","Type":"ContainerDied","Data":"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066"} Oct 30 10:00:30.487248 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.487147 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl" event={"ID":"15fc342a-b869-45c4-9b57-8fce99b62bf6","Type":"ContainerDied","Data":"9c6e4f5c9f3ff222b83cd81862be3e53113679a8efcf7e98d8a6f7c02c3e8912"} Oct 30 10:00:30.487248 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.487178 4920 scope.go:122] "RemoveContainer" containerID="f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066" Oct 30 10:00:30.497544 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.497523 4920 scope.go:122] "RemoveContainer" containerID="f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066" Oct 30 10:00:30.497848 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:30.497826 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066\": container with ID starting with f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066 not found: ID does not exist" containerID="f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066" Oct 30 10:00:30.497936 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.497857 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066"} err="failed to get container status \"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066\": rpc error: code = NotFound desc = could not find container \"f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066\": container with ID starting with f1d314dddaa888862cd4dcaed0a1b6d1824e2e21f7b71b38ce250d4ac3ffb066 not found: ID does not exist" Oct 30 10:00:30.537750 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.537712 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 10:00:30.549181 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.549142 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f58f32-57f476658-x7xnl"] Oct 30 10:00:30.890806 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:30.890750 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:31.137656 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:31.137620 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:00:31.963673 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:31.963636 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:32.397913 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:32.397881 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fc342a-b869-45c4-9b57-8fce99b62bf6" path="/var/lib/kubelet/pods/15fc342a-b869-45c4-9b57-8fce99b62bf6/volumes" Oct 30 10:00:33.270591 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:33.270551 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:00:33.459860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:33.459828 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:35.083007 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:35.082971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:36.647008 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:36.646971 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:36.864808 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:36.864777 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:36.969110 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:36.969023 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:37.195283 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:37.195245 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 10:00:37.195530 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:37.195501 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" containerID="cri-o://6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae" gracePeriod=30 Oct 30 10:00:37.216973 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:37.216938 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:38.179815 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:38.179777 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:39.858815 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:39.858780 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:41.446072 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:41.446022 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:42.469242 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.469152 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 10:00:42.558385 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558352 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558585 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558396 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558585 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558420 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558585 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558463 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558585 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558540 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558850 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558592 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558850 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558651 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558850 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558705 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:42.558850 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558737 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.558850 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558777 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\") pod \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\" (UID: \"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e\") " Oct 30 10:00:42.559121 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558865 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:42.559121 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.558948 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:42.559264 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.559171 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:42.559264 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.559197 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.559264 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.559235 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.559264 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.559255 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.559567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.559545 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:42.561059 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.561032 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:42.561293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.561261 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:42.561378 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.561305 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "kube-api-access-gpf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:42.561440 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.561395 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo" pod "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" (UID: "d16b04fc-2c0a-40b9-ad91-78ccf37aa54e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:42.578062 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.578030 4920 generic.go:356] "Generic (PLEG): container finished" podID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerID="6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae" exitCode=0 Oct 30 10:00:42.578207 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.578079 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerDied","Data":"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae"} Oct 30 10:00:42.578207 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.578140 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" event={"ID":"d16b04fc-2c0a-40b9-ad91-78ccf37aa54e","Type":"ContainerDied","Data":"fc821dbb02e4f77d845802a521590056815a7202bd1ef474609f4bd4e7316789"} Oct 30 10:00:42.578207 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.578083 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz" Oct 30 10:00:42.578207 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.578171 4920 scope.go:122] "RemoveContainer" containerID="6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae" Oct 30 10:00:42.588885 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.588866 4920 scope.go:122] "RemoveContainer" containerID="c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981" Oct 30 10:00:42.598050 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.598028 4920 scope.go:122] "RemoveContainer" containerID="6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae" Oct 30 10:00:42.598372 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:42.598349 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae\": container with ID starting with 6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae not found: ID does not exist" containerID="6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae" Oct 30 10:00:42.598467 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.598387 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae"} err="failed to get container status \"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae\": rpc error: code = NotFound desc = could not find container \"6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae\": container with ID starting with 6cc8e24668d8ba42fe4346a6ad0932e5e3cd2d9806aa526c29d70b4894594dae not found: ID does not exist" Oct 30 10:00:42.598467 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.598413 4920 scope.go:122] "RemoveContainer" containerID="c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981" Oct 30 10:00:42.598704 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:42.598685 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981\": container with ID starting with c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981 not found: ID does not exist" containerID="c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981" Oct 30 10:00:42.598748 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.598709 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981"} err="failed to get container status \"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981\": rpc error: code = NotFound desc = could not find container \"c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981\": container with ID starting with c1501813c096c08c057afc55bc17c094fa1cc1a719e91fd386339732966d1981 not found: ID does not exist" Oct 30 10:00:42.628661 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.628625 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 10:00:42.632891 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.632850 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f58f32-openshift-default-bd8b77d5c-c2bpz"] Oct 30 10:00:42.659866 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659837 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.659866 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659864 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-kube-api-access-gpf56\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.659866 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659876 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.660166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659885 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.660166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659894 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:42.660166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:42.659901 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:43.047072 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:43.047034 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:44.397413 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:44.397369 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" path="/var/lib/kubelet/pods/d16b04fc-2c0a-40b9-ad91-78ccf37aa54e/volumes" Oct 30 10:00:44.551499 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:44.551460 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:46.054910 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:46.054872 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:47.652741 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:47.652703 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:48.363597 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:48.363557 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:48.534819 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:48.534784 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:48.900241 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:48.899551 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 10:00:48.908285 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:48.901716 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" podUID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" containerName="maas-api" containerID="cri-o://14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4" gracePeriod=30 Oct 30 10:00:48.958293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:48.958251 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:49.223751 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.223722 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 10:00:49.331502 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.331468 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\") pod \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " Oct 30 10:00:49.331689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.331533 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\") pod \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " Oct 30 10:00:49.331689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.331595 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\" (UniqueName: \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\") pod \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " Oct 30 10:00:49.331689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.331634 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\") pod \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\" (UID: \"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2\") " Oct 30 10:00:49.332015 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.331975 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca" pod "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" (UID: "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:49.333843 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.333812 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh" pod "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" (UID: "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2"). InnerVolumeSpecName "kube-api-access-qtxwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:49.333843 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.333832 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls" pod "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" (UID: "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:49.334305 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.334283 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs" pod "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" (UID: "5ea5bd4d-d46f-4b4e-8227-da9b85c43af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:00:49.433362 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.433324 4920 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-openshift-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:49.433362 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.433356 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxwh\" (UniqueName: \"kubernetes.io/projected/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-kube-api-access-qtxwh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:49.433362 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.433368 4920 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-metrics-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:49.433618 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.433378 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2-maas-api-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:00:49.557837 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.557782 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:49.633534 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.633499 4920 generic.go:356] "Generic (PLEG): container finished" podID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" containerID="14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4" exitCode=0 Oct 30 10:00:49.633739 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.633557 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" Oct 30 10:00:49.633739 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.633571 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" event={"ID":"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2","Type":"ContainerDied","Data":"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4"} Oct 30 10:00:49.633739 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.633631 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt" event={"ID":"5ea5bd4d-d46f-4b4e-8227-da9b85c43af2","Type":"ContainerDied","Data":"7e57e48609df085b43b078202a89c32c98500405ed7be9e3a038fae0605f4687"} Oct 30 10:00:49.633739 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.633659 4920 scope.go:122] "RemoveContainer" containerID="14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4" Oct 30 10:00:49.646767 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.646737 4920 scope.go:122] "RemoveContainer" containerID="14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4" Oct 30 10:00:49.647099 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:00:49.647076 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4\": container with ID starting with 14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4 not found: ID does not exist" containerID="14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4" Oct 30 10:00:49.647160 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.647108 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4"} err="failed to get container status \"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4\": rpc error: code = NotFound desc = could not find container \"14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4\": container with ID starting with 14f3199240f2cdd2a30de1547d3392711e80c49503a54fc6812db87a1ddd78e4 not found: ID does not exist" Oct 30 10:00:49.686676 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.686637 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 10:00:49.688933 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:49.688907 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8ebd65-6c5959d7bf-7rnzt"] Oct 30 10:00:50.397770 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:50.397736 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" path="/var/lib/kubelet/pods/5ea5bd4d-d46f-4b4e-8227-da9b85c43af2/volumes" Oct 30 10:00:51.060384 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:51.060351 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:52.284094 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:52.284050 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:00:52.662894 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:52.662803 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:54.437311 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:54.437255 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:54.537992 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:54.537946 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:54.640532 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:54.640483 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:54.748251 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:54.748135 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:54.968363 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:54.968322 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:55.115295 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:55.115239 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:56.144359 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:56.144307 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:57.760211 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:57.760136 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:00:59.264123 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:00:59.264080 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:00.893525 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:00.893492 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:01.135421 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:01.135382 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:01:01.812794 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:01.812751 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:01.913475 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:01.913435 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:02.135612 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.135517 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 10:01:02.135814 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.135758 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" containerID="cri-o://af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d" gracePeriod=30 Oct 30 10:01:02.154889 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.154856 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:02.564091 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.564052 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:02.762848 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.762810 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56"] Oct 30 10:01:02.763270 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763256 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" containerName="maas-api" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763271 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" containerName="maas-api" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763284 4920 cpu_manager.go:414] "containerMap: removing container" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763289 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763296 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763302 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763310 4920 cpu_manager.go:414] "containerMap: removing container" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763314 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763324 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:02.763331 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763328 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763345 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763350 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763360 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763365 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763371 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763375 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763382 4920 cpu_manager.go:414] "containerMap: removing container" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763386 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763401 4920 cpu_manager.go:414] "containerMap: removing container" podUID="15fc342a-b869-45c4-9b57-8fce99b62bf6" containerName="maas-api" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763405 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fc342a-b869-45c4-9b57-8fce99b62bf6" containerName="maas-api" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763484 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763491 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763498 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763503 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763510 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763517 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763524 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763529 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763536 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fc342a-b869-45c4-9b57-8fce99b62bf6" containerName="maas-api" Oct 30 10:01:02.763621 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.763542 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea5bd4d-d46f-4b4e-8227-da9b85c43af2" containerName="maas-api" Oct 30 10:01:02.767946 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.767929 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.775090 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.774722 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.775090 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.774766 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvq5\" (UniqueName: \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.775090 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.774952 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.782356 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.782323 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 10:01:02.782356 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.782343 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:01:02.782514 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.782365 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 10:01:02.782514 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.782365 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:01:02.783651 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.783629 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56"] Oct 30 10:01:02.875738 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.875650 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.875738 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.875698 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvq5\" (UniqueName: \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.875995 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.875784 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.876464 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.876439 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.877958 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.877938 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:02.894113 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:02.894086 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvq5\" (UniqueName: \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:03.082203 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:03.082162 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:03.383573 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:03.383539 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56"] Oct 30 10:01:03.384130 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:01:03.384067 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e73dbf_f83f_4439_86d5_354a2cf0e036.slice/crio-97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d WatchSource:0}: Error finding container 97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d: Status 404 returned error can't find the container with id 97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d Oct 30 10:01:03.731177 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:03.731136 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" event={"ID":"39e73dbf-f83f-4439-86d5-354a2cf0e036","Type":"ContainerStarted","Data":"6c507a956c6b7cfb465240c44573e8b48714ece448b4f5bc83f7b75cd152a00b"} Oct 30 10:01:03.731356 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:03.731244 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" event={"ID":"39e73dbf-f83f-4439-86d5-354a2cf0e036","Type":"ContainerStarted","Data":"97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d"} Oct 30 10:01:03.768555 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:03.768501 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" podStartSLOduration=1.768487088 podStartE2EDuration="1.768487088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:01:03.767143084 +0000 UTC m=+4027.490570230" watchObservedRunningTime="2026-10-30 10:01:03.768487088 +0000 UTC m=+4027.491914230" Oct 30 10:01:04.206934 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:04.206849 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:04.738101 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:04.738066 4920 generic.go:356] "Generic (PLEG): container finished" podID="39e73dbf-f83f-4439-86d5-354a2cf0e036" containerID="6c507a956c6b7cfb465240c44573e8b48714ece448b4f5bc83f7b75cd152a00b" exitCode=0 Oct 30 10:01:04.738336 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:04.738149 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" event={"ID":"39e73dbf-f83f-4439-86d5-354a2cf0e036","Type":"ContainerDied","Data":"6c507a956c6b7cfb465240c44573e8b48714ece448b4f5bc83f7b75cd152a00b"} Oct 30 10:01:05.742745 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:05.742705 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:05.906143 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:05.906117 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:06.101356 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.101322 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\") pod \"39e73dbf-f83f-4439-86d5-354a2cf0e036\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " Oct 30 10:01:06.101570 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.101373 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\") pod \"39e73dbf-f83f-4439-86d5-354a2cf0e036\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " Oct 30 10:01:06.101570 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.101409 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\" (UniqueName: \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\") pod \"39e73dbf-f83f-4439-86d5-354a2cf0e036\" (UID: \"39e73dbf-f83f-4439-86d5-354a2cf0e036\") " Oct 30 10:01:06.101765 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.101740 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca" pod "39e73dbf-f83f-4439-86d5-354a2cf0e036" (UID: "39e73dbf-f83f-4439-86d5-354a2cf0e036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:06.101863 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.101849 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:06.103530 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.103506 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5" pod "39e73dbf-f83f-4439-86d5-354a2cf0e036" (UID: "39e73dbf-f83f-4439-86d5-354a2cf0e036"). InnerVolumeSpecName "kube-api-access-9lvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:06.103645 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.103559 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca" pod "39e73dbf-f83f-4439-86d5-354a2cf0e036" (UID: "39e73dbf-f83f-4439-86d5-354a2cf0e036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:06.202846 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.202801 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e73dbf-f83f-4439-86d5-354a2cf0e036-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:06.202846 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.202841 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvq5\" (UniqueName: \"kubernetes.io/projected/39e73dbf-f83f-4439-86d5-354a2cf0e036-kube-api-access-9lvq5\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:06.751954 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.751871 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" Oct 30 10:01:06.751954 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.751887 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56" event={"ID":"39e73dbf-f83f-4439-86d5-354a2cf0e036","Type":"ContainerDied","Data":"97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d"} Oct 30 10:01:06.751954 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.751936 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b35846c9561eae8f46d07c03c7af93a3a4b5c7fd065c915bc67063a33da47d" Oct 30 10:01:06.924965 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.924923 4920 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 10:01:06.925140 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:06.924982 4920 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 10:01:07.256828 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.256792 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:07.406254 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.406179 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 10:01:07.413747 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.413717 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.413881 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.413800 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.413881 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.413867 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.413994 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.413931 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.413994 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.413971 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.414125 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414022 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.414125 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414051 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.414125 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414078 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.414125 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414107 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:07.414427 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414115 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\") pod \"98559665-d64b-4d4f-8787-e3788e6cef75\" (UID: \"98559665-d64b-4d4f-8787-e3788e6cef75\") " Oct 30 10:01:07.414427 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414245 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:07.414536 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414481 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:07.414536 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414502 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:07.414638 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414619 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.414689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414647 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.414689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414669 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.414689 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414684 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98559665-d64b-4d4f-8787-e3788e6cef75-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.414821 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.414762 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:07.416501 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.416470 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:01:07.416609 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.416520 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:07.416675 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.416599 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:07.416675 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.416612 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z" pod "98559665-d64b-4d4f-8787-e3788e6cef75" (UID: "98559665-d64b-4d4f-8787-e3788e6cef75"). InnerVolumeSpecName "kube-api-access-rqz2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:07.515928 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.515895 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.515928 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.515924 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.515928 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.515933 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98559665-d64b-4d4f-8787-e3788e6cef75-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.516166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.515942 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz2z\" (UniqueName: \"kubernetes.io/projected/98559665-d64b-4d4f-8787-e3788e6cef75-kube-api-access-rqz2z\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.516166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.515953 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98559665-d64b-4d4f-8787-e3788e6cef75-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:07.760018 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.759922 4920 generic.go:356] "Generic (PLEG): container finished" podID="98559665-d64b-4d4f-8787-e3788e6cef75" containerID="af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d" exitCode=0 Oct 30 10:01:07.760018 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.759973 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" Oct 30 10:01:07.760573 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.760010 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerDied","Data":"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d"} Oct 30 10:01:07.760573 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.760065 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw" event={"ID":"98559665-d64b-4d4f-8787-e3788e6cef75","Type":"ContainerDied","Data":"3c8ec112b30b1e57fca7d5e19ff5333b58c0ca35ee9d8fde9ad91c749c313b14"} Oct 30 10:01:07.760573 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.760087 4920 scope.go:122] "RemoveContainer" containerID="af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d" Oct 30 10:01:07.771104 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.771082 4920 scope.go:122] "RemoveContainer" containerID="a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba" Oct 30 10:01:07.780355 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.780335 4920 scope.go:122] "RemoveContainer" containerID="af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d" Oct 30 10:01:07.780709 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:07.780680 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d\": container with ID starting with af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d not found: ID does not exist" containerID="af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d" Oct 30 10:01:07.780786 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.780716 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d"} err="failed to get container status \"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d\": rpc error: code = NotFound desc = could not find container \"af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d\": container with ID starting with af328618d3f7588d9e0ac9d07c75850acf2e177bec4847ac48eb367fa6db206d not found: ID does not exist" Oct 30 10:01:07.780786 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.780735 4920 scope.go:122] "RemoveContainer" containerID="a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba" Oct 30 10:01:07.781005 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:07.780985 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba\": container with ID starting with a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba not found: ID does not exist" containerID="a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba" Oct 30 10:01:07.781047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.781011 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba"} err="failed to get container status \"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba\": rpc error: code = NotFound desc = could not find container \"a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba\": container with ID starting with a2ed5109d9fb9be0c6a690929090314f79cad4b7ac9d71b75267ad69073839ba not found: ID does not exist" Oct 30 10:01:07.814813 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.814775 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 10:01:07.826946 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:07.826905 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8ebd65-openshift-default-6469db79bc-7kwcw"] Oct 30 10:01:08.397205 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:08.397167 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" path="/var/lib/kubelet/pods/98559665-d64b-4d4f-8787-e3788e6cef75/volumes" Oct 30 10:01:08.855282 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:08.855235 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:10.448138 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:10.448094 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:12.049826 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:12.049789 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:13.329140 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:13.329088 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:13.420264 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:13.420195 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:13.943212 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:13.943178 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:14.561064 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:14.561018 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:14.661551 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:14.661518 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:14.681160 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:14.681122 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56"] Oct 30 10:01:14.691600 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:14.691568 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-388653-zrz56"] Oct 30 10:01:15.167186 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:15.167150 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:15.479477 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:15.479370 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:16.397688 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:16.397652 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e73dbf-f83f-4439-86d5-354a2cf0e036" path="/var/lib/kubelet/pods/39e73dbf-f83f-4439-86d5-354a2cf0e036/volumes" Oct 30 10:01:17.061856 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:17.061823 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:17.482764 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:17.482683 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:01:17.911513 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:17.911471 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:18.656094 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:18.656062 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:20.158961 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:20.158908 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:20.961854 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:20.961821 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:21.061837 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.061802 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:21.292059 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.292022 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 10:01:21.293669 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.293634 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" containerID="cri-o://3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea" gracePeriod=30 Oct 30 10:01:21.320470 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.320425 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:21.763672 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.763567 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:21.863881 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:21.863850 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:22.043934 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.043904 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p"] Oct 30 10:01:22.044451 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044436 4920 cpu_manager.go:414] "containerMap: removing container" podUID="39e73dbf-f83f-4439-86d5-354a2cf0e036" containerName="revoke-keys" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044452 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e73dbf-f83f-4439-86d5-354a2cf0e036" containerName="revoke-keys" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044464 4920 cpu_manager.go:414] "containerMap: removing container" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044469 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044484 4920 cpu_manager.go:414] "containerMap: removing container" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044492 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044506 4920 cpu_manager.go:414] "containerMap: removing container" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044510 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044529 4920 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:22.044531 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044534 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea94c2-b95f-4263-8fcb-56e1867ef1ba" containerName="cleanup" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044546 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044551 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044628 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e73dbf-f83f-4439-86d5-354a2cf0e036" containerName="revoke-keys" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044638 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044644 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="37683065-673f-4bf6-bc1b-6424ad8c7765" containerName="cleanup" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044651 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16b04fc-2c0a-40b9-ad91-78ccf37aa54e" containerName="istio-proxy" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044657 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="98559665-d64b-4d4f-8787-e3788e6cef75" containerName="istio-proxy" Oct 30 10:01:22.044860 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.044666 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f90dfa0-6bbe-4348-ae10-17ad14fec97f" containerName="cleanup" Oct 30 10:01:22.049367 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.049347 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.064060 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.064032 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:01:22.064470 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.064446 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:01:22.064596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.064516 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 10:01:22.064596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.064533 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 10:01:22.066628 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.066600 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p"] Oct 30 10:01:22.156519 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.156476 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.156743 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.156548 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4sd\" (UniqueName: \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.156743 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.156621 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.257934 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.257892 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.258142 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.257993 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.258142 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.258028 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4sd\" (UniqueName: \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.258749 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.258724 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.260443 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.260414 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.276636 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.276609 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4sd\" (UniqueName: \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\") pod \"maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.362641 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.362551 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:22.664144 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:01:22.664107 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db13cce_26a3_488e_8ac9_3e3bc585d505.slice/crio-aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114 WatchSource:0}: Error finding container aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114: Status 404 returned error can't find the container with id aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114 Oct 30 10:01:22.665343 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.665320 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p"] Oct 30 10:01:22.855964 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:22.855923 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" event={"ID":"3db13cce-26a3-488e-8ac9-3e3bc585d505","Type":"ContainerStarted","Data":"aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114"} Oct 30 10:01:23.352806 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:23.352778 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:23.864159 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:23.864123 4920 generic.go:356] "Generic (PLEG): container finished" podID="3db13cce-26a3-488e-8ac9-3e3bc585d505" containerID="3f0a673e35891e82fe560d102ad2092fd7c5a6c1dcc31702cd3fc8e297237af0" exitCode=0 Oct 30 10:01:23.864587 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:23.864203 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" event={"ID":"3db13cce-26a3-488e-8ac9-3e3bc585d505","Type":"ContainerDied","Data":"3f0a673e35891e82fe560d102ad2092fd7c5a6c1dcc31702cd3fc8e297237af0"} Oct 30 10:01:24.951435 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:24.951396 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:25.018679 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.018641 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:25.086467 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.086432 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\") pod \"3db13cce-26a3-488e-8ac9-3e3bc585d505\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " Oct 30 10:01:25.086657 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.086498 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\") pod \"3db13cce-26a3-488e-8ac9-3e3bc585d505\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " Oct 30 10:01:25.086657 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.086556 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\" (UniqueName: \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\") pod \"3db13cce-26a3-488e-8ac9-3e3bc585d505\" (UID: \"3db13cce-26a3-488e-8ac9-3e3bc585d505\") " Oct 30 10:01:25.086919 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.086880 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca" pod "3db13cce-26a3-488e-8ac9-3e3bc585d505" (UID: "3db13cce-26a3-488e-8ac9-3e3bc585d505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:25.088773 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.088751 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca" pod "3db13cce-26a3-488e-8ac9-3e3bc585d505" (UID: "3db13cce-26a3-488e-8ac9-3e3bc585d505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:25.088880 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.088778 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd" pod "3db13cce-26a3-488e-8ac9-3e3bc585d505" (UID: "3db13cce-26a3-488e-8ac9-3e3bc585d505"). InnerVolumeSpecName "kube-api-access-bp4sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:25.187596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.187501 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:25.187596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.187539 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db13cce-26a3-488e-8ac9-3e3bc585d505-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:25.187596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.187553 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp4sd\" (UniqueName: \"kubernetes.io/projected/3db13cce-26a3-488e-8ac9-3e3bc585d505-kube-api-access-bp4sd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:25.877455 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.877421 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" Oct 30 10:01:25.877644 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.877455 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p" event={"ID":"3db13cce-26a3-488e-8ac9-3e3bc585d505","Type":"ContainerDied","Data":"aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114"} Oct 30 10:01:25.877644 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:25.877504 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa87edd074c1f399521cf12756e2699bfa6288f9df52f20c3084fa7243735114" Oct 30 10:01:26.545765 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.545741 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 10:01:26.561054 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.560991 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:26.601897 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.601867 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602099 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.601937 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602099 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.601985 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602099 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602018 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602099 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602053 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602358 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602123 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602358 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602173 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:26.602358 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602176 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602358 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602322 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602376 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\") pod \"c60a5b8e-d331-4901-a58a-77885fe3c002\" (UID: \"c60a5b8e-d331-4901-a58a-77885fe3c002\") " Oct 30 10:01:26.602567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602355 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:26.602567 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602426 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:26.602734 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602696 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:26.602803 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602727 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:26.602803 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602765 4920 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.602803 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602786 4920 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-workload-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.602803 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.602799 4920 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-data\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.605399 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.605341 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:01:26.605399 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.605357 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:26.605399 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.605378 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "kube-api-access-9ktdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:26.607660 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.607640 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token" pod "c60a5b8e-d331-4901-a58a-77885fe3c002" (UID: "c60a5b8e-d331-4901-a58a-77885fe3c002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:26.703414 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703377 4920 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60a5b8e-d331-4901-a58a-77885fe3c002-istiod-ca-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.703414 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703409 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktdj\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-kube-api-access-9ktdj\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.703414 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703419 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.703668 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703432 4920 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-envoy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.703668 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703444 4920 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60a5b8e-d331-4901-a58a-77885fe3c002-istio-podinfo\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.703668 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.703453 4920 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60a5b8e-d331-4901-a58a-77885fe3c002-credential-socket\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:26.788699 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.788661 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 10:01:26.878625 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.878538 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:26.885113 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.885085 4920 generic.go:356] "Generic (PLEG): container finished" podID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerID="3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea" exitCode=0 Oct 30 10:01:26.885293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.885141 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerDied","Data":"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea"} Oct 30 10:01:26.885293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.885147 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" Oct 30 10:01:26.885293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.885249 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw" event={"ID":"c60a5b8e-d331-4901-a58a-77885fe3c002","Type":"ContainerDied","Data":"5760a5a0246816cccae42359af6cbd282050d8d263172f2d54e329b43f947b11"} Oct 30 10:01:26.885293 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.885281 4920 scope.go:122] "RemoveContainer" containerID="3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea" Oct 30 10:01:26.898306 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.898289 4920 scope.go:122] "RemoveContainer" containerID="47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4" Oct 30 10:01:26.907149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.907128 4920 scope.go:122] "RemoveContainer" containerID="3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea" Oct 30 10:01:26.907435 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:26.907415 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea\": container with ID starting with 3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea not found: ID does not exist" containerID="3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea" Oct 30 10:01:26.907491 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.907445 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea"} err="failed to get container status \"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea\": rpc error: code = NotFound desc = could not find container \"3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea\": container with ID starting with 3eb4f2ec88e547db8066ccecb002d15f4f3c823586752e3bd0d4f45bcab282ea not found: ID does not exist" Oct 30 10:01:26.907491 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.907461 4920 scope.go:122] "RemoveContainer" containerID="47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4" Oct 30 10:01:26.907676 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:26.907662 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4\": container with ID starting with 47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4 not found: ID does not exist" containerID="47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4" Oct 30 10:01:26.907722 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.907680 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4"} err="failed to get container status \"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4\": rpc error: code = NotFound desc = could not find container \"47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4\": container with ID starting with 47156f4452fd33cb4c99a42ea4f2a55b3179967578d198e3526be9045e71d1b4 not found: ID does not exist" Oct 30 10:01:26.940077 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.940044 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 10:01:26.946339 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:26.946310 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-388653-openshift-default-6cd774c57f-bjztw"] Oct 30 10:01:28.148889 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:28.148851 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:28.397261 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:28.397199 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" path="/var/lib/kubelet/pods/c60a5b8e-d331-4901-a58a-77885fe3c002/volumes" Oct 30 10:01:29.748447 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:29.748400 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:31.133862 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:31.133828 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:01:31.252969 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:31.252936 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:32.516712 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.516674 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx"] Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517099 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517111 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517120 4920 cpu_manager.go:414] "containerMap: removing container" podUID="3db13cce-26a3-488e-8ac9-3e3bc585d505" containerName="revoke-keys" Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517125 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db13cce-26a3-488e-8ac9-3e3bc585d505" containerName="revoke-keys" Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517139 4920 cpu_manager.go:414] "containerMap: removing container" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.517149 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517145 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.517393 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517231 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.517393 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517238 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db13cce-26a3-488e-8ac9-3e3bc585d505" containerName="revoke-keys" Oct 30 10:01:32.517393 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.517246 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60a5b8e-d331-4901-a58a-77885fe3c002" containerName="istio-proxy" Oct 30 10:01:32.521765 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.521746 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.528281 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.528251 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 10:01:32.537454 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.537398 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 10:01:32.540030 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.539983 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx"] Oct 30 10:01:32.540735 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.540710 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:01:32.540826 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.540715 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 10:01:32.540826 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.540717 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:01:32.561316 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.561282 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 10:01:32.655859 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.655828 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:32.663152 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.663121 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.663327 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.663167 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.663327 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.663269 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhqh\" (UniqueName: \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.762709 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.762654 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:32.764901 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.764861 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhqh\" (UniqueName: \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.765812 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.765074 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.765812 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.765121 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.766245 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.766190 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.767748 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.767695 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.792575 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.792540 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhqh\" (UniqueName: \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\") pod \"maas-api-revoke-sub-76a917c2-jzgrx\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:32.835692 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:32.835662 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:33.137702 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:33.137669 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx"] Oct 30 10:01:33.137976 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:01:33.137948 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661a2bc6_90bb_4f15_bd67_33043c9ebbaf.slice/crio-062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0 WatchSource:0}: Error finding container 062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0: Status 404 returned error can't find the container with id 062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0 Oct 30 10:01:33.932556 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:33.932519 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" event={"ID":"661a2bc6-90bb-4f15-bd67-33043c9ebbaf","Type":"ContainerStarted","Data":"91d2feab5857e27da0ac3b452257c238dd7b74ddf2ee4e7a903c27b81ca803b9"} Oct 30 10:01:33.933024 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:33.932581 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" event={"ID":"661a2bc6-90bb-4f15-bd67-33043c9ebbaf","Type":"ContainerStarted","Data":"062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0"} Oct 30 10:01:33.971920 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:33.971848 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" podStartSLOduration=1.9718307149999998 podStartE2EDuration="1.971830715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:01:33.970737664 +0000 UTC m=+4057.694164806" watchObservedRunningTime="2026-10-30 10:01:33.971830715 +0000 UTC m=+4057.695257860" Oct 30 10:01:34.940136 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:34.940100 4920 generic.go:356] "Generic (PLEG): container finished" podID="661a2bc6-90bb-4f15-bd67-33043c9ebbaf" containerID="91d2feab5857e27da0ac3b452257c238dd7b74ddf2ee4e7a903c27b81ca803b9" exitCode=0 Oct 30 10:01:34.940625 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:34.940162 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" event={"ID":"661a2bc6-90bb-4f15-bd67-33043c9ebbaf","Type":"ContainerDied","Data":"91d2feab5857e27da0ac3b452257c238dd7b74ddf2ee4e7a903c27b81ca803b9"} Oct 30 10:01:36.100177 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.100148 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:36.198271 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.198150 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\") pod \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " Oct 30 10:01:36.198477 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.198279 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\" (UniqueName: \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\") pod \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " Oct 30 10:01:36.198477 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.198334 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\") pod \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\" (UID: \"661a2bc6-90bb-4f15-bd67-33043c9ebbaf\") " Oct 30 10:01:36.198614 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.198557 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca" pod "661a2bc6-90bb-4f15-bd67-33043c9ebbaf" (UID: "661a2bc6-90bb-4f15-bd67-33043c9ebbaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:36.200549 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.200510 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh" pod "661a2bc6-90bb-4f15-bd67-33043c9ebbaf" (UID: "661a2bc6-90bb-4f15-bd67-33043c9ebbaf"). InnerVolumeSpecName "kube-api-access-dlhqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:36.200652 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.200516 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca" pod "661a2bc6-90bb-4f15-bd67-33043c9ebbaf" (UID: "661a2bc6-90bb-4f15-bd67-33043c9ebbaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:36.299275 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.299238 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhqh\" (UniqueName: \"kubernetes.io/projected/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-kube-api-access-dlhqh\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:36.299275 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.299270 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-xks-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:36.299275 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.299280 4920 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661a2bc6-90bb-4f15-bd67-33043c9ebbaf-maas-api-openshift-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:36.952139 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.952100 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" Oct 30 10:01:36.952357 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.952104 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a917c2-jzgrx" event={"ID":"661a2bc6-90bb-4f15-bd67-33043c9ebbaf","Type":"ContainerDied","Data":"062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0"} Oct 30 10:01:36.952357 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:36.952262 4920 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062cbe4d54522459b133a238c280ae1b8f893fe8bb3c30a6e321d020317a14e0" Oct 30 10:01:37.528816 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.528757 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" podUID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" containerName="payload-processing" containerID="cri-o://54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2" gracePeriod=25 Oct 30 10:01:37.562435 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.562369 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" podUID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" containerName="payload-pre-processing" containerID="cri-o://0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da" gracePeriod=25 Oct 30 10:01:37.817745 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.817710 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 10:01:37.850775 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.850741 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 10:01:37.913882 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.913508 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\") pod \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " Oct 30 10:01:37.913882 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.913576 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\" (UniqueName: \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\") pod \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\" (UID: \"5ff9350e-f9cd-4ed6-9b44-6305633eb32e\") " Oct 30 10:01:37.913882 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.913689 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\" (UniqueName: \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\") pod \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " Oct 30 10:01:37.913882 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.913778 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\") pod \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\" (UID: \"328a0d1b-a315-43f9-9bde-c0bdc2503b27\") " Oct 30 10:01:37.914267 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.913913 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume" pod "5ff9350e-f9cd-4ed6-9b44-6305633eb32e" (UID: "5ff9350e-f9cd-4ed6-9b44-6305633eb32e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:37.914338 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.914273 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume" pod "328a0d1b-a315-43f9-9bde-c0bdc2503b27" (UID: "328a0d1b-a315-43f9-9bde-c0bdc2503b27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:37.914823 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.914797 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a0d1b-a315-43f9-9bde-c0bdc2503b27-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:37.914936 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.914826 4920 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:37.916207 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.916182 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn" pod "5ff9350e-f9cd-4ed6-9b44-6305633eb32e" (UID: "5ff9350e-f9cd-4ed6-9b44-6305633eb32e"). InnerVolumeSpecName "kube-api-access-xgkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:37.916307 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.916293 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w" pod "328a0d1b-a315-43f9-9bde-c0bdc2503b27" (UID: "328a0d1b-a315-43f9-9bde-c0bdc2503b27"). InnerVolumeSpecName "kube-api-access-jjx2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:37.962144 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.962112 4920 generic.go:356] "Generic (PLEG): container finished" podID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" containerID="54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2" exitCode=0 Oct 30 10:01:37.962375 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.962274 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" Oct 30 10:01:37.962521 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.962270 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" event={"ID":"328a0d1b-a315-43f9-9bde-c0bdc2503b27","Type":"ContainerDied","Data":"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2"} Oct 30 10:01:37.962521 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.962491 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx" event={"ID":"328a0d1b-a315-43f9-9bde-c0bdc2503b27","Type":"ContainerDied","Data":"ed070c644bf5c15f60308083184099445db244c7ff2624a06f8f218ff0789094"} Oct 30 10:01:37.962631 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.962529 4920 scope.go:122] "RemoveContainer" containerID="54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2" Oct 30 10:01:37.963417 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.963390 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:37.964166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.964122 4920 generic.go:356] "Generic (PLEG): container finished" podID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" containerID="0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da" exitCode=0 Oct 30 10:01:37.964166 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.964151 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" event={"ID":"5ff9350e-f9cd-4ed6-9b44-6305633eb32e","Type":"ContainerDied","Data":"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da"} Oct 30 10:01:37.964309 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.964182 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" Oct 30 10:01:37.964309 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.964189 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd" event={"ID":"5ff9350e-f9cd-4ed6-9b44-6305633eb32e","Type":"ContainerDied","Data":"6baae6623be1814042aa07b9728491fa66ed800a2f4147722dd73eb25f4b902f"} Oct 30 10:01:37.974261 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.974206 4920 scope.go:122] "RemoveContainer" containerID="54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2" Oct 30 10:01:37.974605 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:37.974582 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2\": container with ID starting with 54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2 not found: ID does not exist" containerID="54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2" Oct 30 10:01:37.974790 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.974614 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2"} err="failed to get container status \"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2\": rpc error: code = NotFound desc = could not find container \"54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2\": container with ID starting with 54822317639d5152acff1d9ecc69352f19459a5953db8b368919947e1a9143c2 not found: ID does not exist" Oct 30 10:01:37.974790 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.974637 4920 scope.go:122] "RemoveContainer" containerID="0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da" Oct 30 10:01:37.983455 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.983435 4920 scope.go:122] "RemoveContainer" containerID="0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da" Oct 30 10:01:37.983742 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:01:37.983721 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da\": container with ID starting with 0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da not found: ID does not exist" containerID="0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da" Oct 30 10:01:37.983780 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:37.983751 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da"} err="failed to get container status \"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da\": rpc error: code = NotFound desc = could not find container \"0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da\": container with ID starting with 0753cc7b8d37e8df5a2b0d7a90b7ce94f865de98ba83813bf812db2bc0e424da not found: ID does not exist" Oct 30 10:01:38.013280 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.013241 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 10:01:38.015509 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.015476 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgkjn\" (UniqueName: \"kubernetes.io/projected/5ff9350e-f9cd-4ed6-9b44-6305633eb32e-kube-api-access-xgkjn\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:38.015619 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.015511 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjx2w\" (UniqueName: \"kubernetes.io/projected/328a0d1b-a315-43f9-9bde-c0bdc2503b27-kube-api-access-jjx2w\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:38.017563 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.017542 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2f69c8-66755b489d-m8vfx"] Oct 30 10:01:38.050335 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.050250 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 10:01:38.052463 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.052431 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:38.054092 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.054071 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2f69c8-5c5b8df9dc-xfxrd"] Oct 30 10:01:38.396801 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.396722 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" path="/var/lib/kubelet/pods/328a0d1b-a315-43f9-9bde-c0bdc2503b27/volumes" Oct 30 10:01:38.397169 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:38.397154 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" path="/var/lib/kubelet/pods/5ff9350e-f9cd-4ed6-9b44-6305633eb32e/volumes" Oct 30 10:01:41.809358 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:41.809320 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" containerID="cri-o://d31dbe5ed19a87aeb019368acd547d2878b6e17fac88207f7c50e2d55e959833" gracePeriod=45 Oct 30 10:01:41.992031 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:41.991997 4920 generic.go:356] "Generic (PLEG): container finished" podID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerID="d31dbe5ed19a87aeb019368acd547d2878b6e17fac88207f7c50e2d55e959833" exitCode=0 Oct 30 10:01:41.992210 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:41.992077 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" event={"ID":"25e769a8-fd84-47f3-b5e1-b44328f557bd","Type":"ContainerDied","Data":"d31dbe5ed19a87aeb019368acd547d2878b6e17fac88207f7c50e2d55e959833"} Oct 30 10:01:42.071260 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.071161 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 10:01:42.156142 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156105 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\" (UniqueName: \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156368 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156247 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156368 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156310 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156508 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156378 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156508 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156422 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156508 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156471 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\") pod \"25e769a8-fd84-47f3-b5e1-b44328f557bd\" (UID: \"25e769a8-fd84-47f3-b5e1-b44328f557bd\") " Oct 30 10:01:42.156663 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156555 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.156800 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156772 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.156800 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156791 4920 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-model-cache\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.156931 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.156786 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.159125 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.159096 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.159270 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.159144 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:42.159270 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.159149 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2" pod "25e769a8-fd84-47f3-b5e1-b44328f557bd" (UID: "25e769a8-fd84-47f3-b5e1-b44328f557bd"). InnerVolumeSpecName "kube-api-access-kk2p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:42.258095 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.258060 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2p2\" (UniqueName: \"kubernetes.io/projected/25e769a8-fd84-47f3-b5e1-b44328f557bd-kube-api-access-kk2p2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.258095 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.258092 4920 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e769a8-fd84-47f3-b5e1-b44328f557bd-tls-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.258095 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.258102 4920 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-dshm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.258372 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.258110 4920 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-home\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.258372 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:42.258119 4920 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e769a8-fd84-47f3-b5e1-b44328f557bd-tmp-dir\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:01:43.000321 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:43.000281 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" Oct 30 10:01:43.000971 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:43.000271 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z" event={"ID":"25e769a8-fd84-47f3-b5e1-b44328f557bd","Type":"ContainerDied","Data":"1d86fec4456c7d3cf93d2265aaa6c38b019387988a31e78f5aeb5d1a437e7cc2"} Oct 30 10:01:43.000971 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:43.000438 4920 scope.go:122] "RemoveContainer" containerID="d31dbe5ed19a87aeb019368acd547d2878b6e17fac88207f7c50e2d55e959833" Oct 30 10:01:43.049565 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:43.049529 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 10:01:43.052638 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:43.052614 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-w0-2f69c8-kserve-ccd7b6b8c-btf4z"] Oct 30 10:01:44.397498 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:44.397460 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" path="/var/lib/kubelet/pods/25e769a8-fd84-47f3-b5e1-b44328f557bd/volumes" Oct 30 10:01:45.534708 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:45.534662 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:01:47.218561 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:47.218523 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:47.295787 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:47.295749 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:47.669201 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:47.669169 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:48.107790 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.107751 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:48.207390 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.207358 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:48.233379 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.233347 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p"] Oct 30 10:01:48.246156 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.246126 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2f69c8-jld8p"] Oct 30 10:01:48.397502 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.397426 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db13cce-26a3-488e-8ac9-3e3bc585d505" path="/var/lib/kubelet/pods/3db13cce-26a3-488e-8ac9-3e3bc585d505/volumes" Oct 30 10:01:48.731180 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:48.731100 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:54.294888 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:54.294851 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:54.395333 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:54.395289 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:54.659145 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:54.659068 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:54.871780 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:54.871748 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:01:56.049849 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:56.049812 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dscb_97ce6c5e-2a90-43bb-8c99-57a28daf5338/manager/0.log" Oct 30 10:01:56.166049 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:56.166008 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vzh5l"] Oct 30 10:01:56.274237 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:56.274181 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-slvzx_20210cd6-62e9-436c-bc29-84bc56bb2589/manager/0.log" Oct 30 10:01:56.480735 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:56.480663 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwnjf_ca4979b8-86d1-4bc9-b06b-ad36059c1279/kuadrant-console-plugin/0.log" Oct 30 10:01:57.297497 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:57.297457 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n98s2_04274227-f7ec-4b45-b118-958c96a7b20c/manager/0.log" Oct 30 10:01:57.500786 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:57.500745 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vzh5l_61f8350c-76c3-464d-9910-c7be862e5628/limitador/0.log" Oct 30 10:01:57.699368 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:57.699291 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4wqhg_753f3050-bd21-4575-997c-157765af79b3/manager/0.log" Oct 30 10:01:58.728951 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:58.728920 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:01:58.929945 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:58.929912 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:01:59.840596 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:01:59.840566 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:01.135234 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:02:01.135178 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:02:03.248172 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:03.248138 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:03.348160 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:03.348125 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:04.348541 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:04.348509 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:09.808604 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:09.808560 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:12.141176 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:12.141135 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:02:12.359059 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:12.359023 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:02:13.262442 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:13.262413 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:15.510450 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:15.510422 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-76a917c2-jzgrx_661a2bc6-90bb-4f15-bd67-33043c9ebbaf/revoke-keys/0.log" Oct 30 10:02:15.930918 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:15.930886 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-af9ebb7f-tws9b_a61ebbd7-389b-4756-b6a0-ebcd1bb16f34/revoke-keys/0.log" Oct 30 10:02:18.581537 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:18.581504 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dscb_97ce6c5e-2a90-43bb-8c99-57a28daf5338/manager/0.log" Oct 30 10:02:18.812688 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:18.812651 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-slvzx_20210cd6-62e9-436c-bc29-84bc56bb2589/manager/0.log" Oct 30 10:02:19.007383 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:19.007298 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwnjf_ca4979b8-86d1-4bc9-b06b-ad36059c1279/kuadrant-console-plugin/0.log" Oct 30 10:02:19.690581 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:19.690542 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n98s2_04274227-f7ec-4b45-b118-958c96a7b20c/manager/0.log" Oct 30 10:02:19.923528 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:19.923499 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vzh5l_61f8350c-76c3-464d-9910-c7be862e5628/limitador/0.log" Oct 30 10:02:20.131135 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:20.131097 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4wqhg_753f3050-bd21-4575-997c-157765af79b3/manager/0.log" Oct 30 10:02:20.914499 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:20.914464 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-lndw2_8218b15b-003a-4f9c-bed8-c74889c32f46/istio-proxy/0.log" Oct 30 10:02:21.117587 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:21.117560 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97vfn_2ccda3a0-e93c-4d5b-a694-51e310cab997/discovery/0.log" Oct 30 10:02:22.775046 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:22.775012 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-z896s_d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c/istio-proxy/1.log" Oct 30 10:02:22.989377 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:22.989348 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-c6v55_5653d03d-10e3-41a3-81ec-275974428909/istio-proxy/0.log" Oct 30 10:02:23.599021 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:23.598992 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7888d9c8bc-nzrhq_f486f2e8-4167-421a-8f04-5f6ce10b6c25/router/0.log" Oct 30 10:02:24.057820 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:24.057788 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-z896s_d7e849f1-8b29-4f70-a8fc-a9b817a1ed7c/istio-proxy/0.log" Oct 30 10:02:25.225712 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:25.225677 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-56jtq_ab135bfc-97aa-4258-8e8a-5375fa34181d/main/0.log" Oct 30 10:02:25.820971 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:25.820941 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rrfbf_81d12f5f-cf69-485c-af8d-b781fcacf16b/main/0.log" Oct 30 10:02:26.025625 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:26.025594 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ftp6n_46e36101-1742-43d6-82dd-010f3d70bcf1/main/0.log" Oct 30 10:02:31.135528 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:02:31.135492 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:02:36.371786 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:36.371751 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:36.485754 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:36.485718 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:43.074878 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:43.074850 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p9764_7d7f6695-43a2-4f59-83c5-ee24748be7f5/global-pull-secret-syncer/0.log" Oct 30 10:02:43.410679 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:43.410604 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8lfmm_627ed97e-e994-4439-815e-d7fe229402e5/connection-checker/0.log" Oct 30 10:02:43.537633 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:43.537604 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ccr66_0d9c9c95-575e-4d45-9416-2c8bd643a5e3/connection-checker/0.log" Oct 30 10:02:43.665519 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:43.665443 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hhlxw_fe9b72a7-a9b7-4fbb-b3ec-248a6f95c169/connection-checker/0.log" Oct 30 10:02:43.828681 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:43.828648 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" Oct 30 10:02:44.266298 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:44.266263 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-105.ec2.internal_ba69e635f74a36f196d077f731ece3db/haproxy/0.log" Oct 30 10:02:45.682317 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:45.682286 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:02:45.733277 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:45.733243 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:02:45.784054 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:45.784023 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:02:45.830397 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:45.830360 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:02:48.384040 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:48.384013 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:48.407970 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:48.407944 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:51.714832 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:51.714803 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dscb_97ce6c5e-2a90-43bb-8c99-57a28daf5338/manager/0.log" Oct 30 10:02:51.841135 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:51.841101 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-slvzx_20210cd6-62e9-436c-bc29-84bc56bb2589/manager/0.log" Oct 30 10:02:51.967424 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:51.967349 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwnjf_ca4979b8-86d1-4bc9-b06b-ad36059c1279/kuadrant-console-plugin/0.log" Oct 30 10:02:52.992575 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.992535 4920 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks"] Oct 30 10:02:52.992996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.992981 4920 cpu_manager.go:414] "containerMap: removing container" podUID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" containerName="payload-processing" Oct 30 10:02:52.993047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.992999 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" containerName="payload-processing" Oct 30 10:02:52.993047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993009 4920 cpu_manager.go:414] "containerMap: removing container" podUID="661a2bc6-90bb-4f15-bd67-33043c9ebbaf" containerName="revoke-keys" Oct 30 10:02:52.993047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993015 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661a2bc6-90bb-4f15-bd67-33043c9ebbaf" containerName="revoke-keys" Oct 30 10:02:52.993047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993035 4920 cpu_manager.go:414] "containerMap: removing container" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" Oct 30 10:02:52.993047 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993040 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993056 4920 cpu_manager.go:414] "containerMap: removing container" podUID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" containerName="payload-pre-processing" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993060 4920 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" containerName="payload-pre-processing" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993119 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="328a0d1b-a315-43f9-9bde-c0bdc2503b27" containerName="payload-processing" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993129 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="661a2bc6-90bb-4f15-bd67-33043c9ebbaf" containerName="revoke-keys" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993138 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e769a8-fd84-47f3-b5e1-b44328f557bd" containerName="main" Oct 30 10:02:52.993233 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.993145 4920 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff9350e-f9cd-4ed6-9b44-6305633eb32e" containerName="payload-pre-processing" Oct 30 10:02:52.996426 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:52.996407 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.017519 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.017486 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks"] Oct 30 10:02:53.023816 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.023781 4920 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6llqt\"/\"default-dockercfg-tfjr2\"" Oct 30 10:02:53.023816 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.023806 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"openshift-service-ca.crt\"" Oct 30 10:02:53.024059 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.023976 4920 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"kube-root-ca.crt\"" Oct 30 10:02:53.145245 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.145189 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-proc\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.145434 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.145350 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-podres\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.145434 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.145389 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7j7\" (UniqueName: \"kubernetes.io/projected/a676795d-4100-48f3-9d6c-2a087472e130-kube-api-access-wn7j7\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.145517 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.145483 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-lib-modules\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.145559 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.145523 4920 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-sys\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246098 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246000 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-proc\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246098 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246071 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-podres\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246098 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246094 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7j7\" (UniqueName: \"kubernetes.io/projected/a676795d-4100-48f3-9d6c-2a087472e130-kube-api-access-wn7j7\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246133 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-lib-modules\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246151 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-proc\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246168 4920 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-sys\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246202 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-podres\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246260 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-sys\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.246410 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.246260 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a676795d-4100-48f3-9d6c-2a087472e130-lib-modules\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.265638 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.265605 4920 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7j7\" (UniqueName: \"kubernetes.io/projected/a676795d-4100-48f3-9d6c-2a087472e130-kube-api-access-wn7j7\") pod \"perf-node-gather-daemonset-xc9ks\" (UID: \"a676795d-4100-48f3-9d6c-2a087472e130\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.291480 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.291449 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:02:53.309367 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.309330 4920 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:53.368136 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.368113 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n98s2_04274227-f7ec-4b45-b118-958c96a7b20c/manager/0.log" Oct 30 10:02:53.372065 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.372035 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:02:53.402902 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.402871 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:02:53.469919 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.469884 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:02:53.608585 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.608553 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vzh5l_61f8350c-76c3-464d-9910-c7be862e5628/limitador/0.log" Oct 30 10:02:53.614186 ip-10-0-129-105 kubenswrapper[4920]: W1030 10:02:53.614153 4920 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda676795d_4100_48f3_9d6c_2a087472e130.slice/crio-4b263eb894e9aeb2ed4015acd5d44355ca018759a81581f2b848104382dea8e7 WatchSource:0}: Error finding container 4b263eb894e9aeb2ed4015acd5d44355ca018759a81581f2b848104382dea8e7: Status 404 returned error can't find the container with id 4b263eb894e9aeb2ed4015acd5d44355ca018759a81581f2b848104382dea8e7 Oct 30 10:02:53.614325 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.614185 4920 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks"] Oct 30 10:02:53.764764 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:53.764724 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4wqhg_753f3050-bd21-4575-997c-157765af79b3/manager/0.log" Oct 30 10:02:54.428888 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:54.428847 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" event={"ID":"a676795d-4100-48f3-9d6c-2a087472e130","Type":"ContainerStarted","Data":"519452ed47a08e59e9fb1e4e05bc6453ad7ba0eae165b95deca6f37eb93847b6"} Oct 30 10:02:54.429316 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:54.428913 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:02:54.429316 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:54.428938 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" event={"ID":"a676795d-4100-48f3-9d6c-2a087472e130","Type":"ContainerStarted","Data":"4b263eb894e9aeb2ed4015acd5d44355ca018759a81581f2b848104382dea8e7"} Oct 30 10:02:54.467940 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:54.467887 4920 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" podStartSLOduration=2.4678736470000002 podStartE2EDuration="2.467873647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:54.467071297 +0000 UTC m=+4138.190498440" watchObservedRunningTime="2026-10-30 10:02:54.467873647 +0000 UTC m=+4138.191300789" Oct 30 10:02:54.996644 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:54.996611 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:02:55.550443 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:55.550407 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:55.611718 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:55.611689 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:02:55.814693 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:55.814613 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-stsh2_2d50e209-0bda-4891-be9d-9e3580efab4e/cluster-monitoring-operator/0.log" Oct 30 10:02:56.905044 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:56.905017 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frr44_c577300a-68a1-4849-99c2-d0af317d1432/node-exporter/0.log" Oct 30 10:02:57.025697 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:57.025669 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frr44_c577300a-68a1-4849-99c2-d0af317d1432/kube-rbac-proxy/0.log" Oct 30 10:02:57.144347 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:02:57.144317 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frr44_c577300a-68a1-4849-99c2-d0af317d1432/init-textfile/0.log" Oct 30 10:03:00.452448 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:00.452414 4920 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-xc9ks" Oct 30 10:03:00.556086 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:00.556054 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:03:00.654191 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:00.654161 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28h4p_eaf9b82c-6b5f-454c-a452-747042200b5b/manager/0.log" Oct 30 10:03:00.659281 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:00.659255 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:03:00.782800 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:00.782711 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b6z6c_1c7a1f0e-704f-4653-9cfc-2a1aef054240/manager/0.log" Oct 30 10:03:01.134764 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:03:01.134731 4920 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sxh78" containerName="extractor" Oct 30 10:03:01.757740 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:01.757706 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-4p7wn_e95cb918-673c-4d8d-ac5e-3b8b2c38bab8/networking-console-plugin/0.log" Oct 30 10:03:02.967490 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:02.967459 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:03:03.013287 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:03.013253 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/3.log" Oct 30 10:03:03.016708 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:03.016691 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mxzf_adbbc457-38a1-45fd-ab3d-738f7dd07913/console-operator/4.log" Oct 30 10:03:03.058031 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:03.058005 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mpftf_9d00f523-413e-4a63-800f-1a813a802d36/server/0.log" Oct 30 10:03:05.176996 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:05.176963 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-8jhj4_412a8656-9cb2-4e67-9752-f179cd02219b/volume-data-source-validator/0.log" Oct 30 10:03:06.299055 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:06.299021 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-lndw2_8218b15b-003a-4f9c-bed8-c74889c32f46/istio-proxy/0.log" Oct 30 10:03:08.081987 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.081950 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mj9w5_c9e051aa-16cd-43b2-aec2-4f19f35ab485/dns/0.log" Oct 30 10:03:08.129089 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.129027 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-lndw2_8218b15b-003a-4f9c-bed8-c74889c32f46/istio-proxy/0.log" Oct 30 10:03:08.141400 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.141373 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-lndw2_8218b15b-003a-4f9c-bed8-c74889c32f46/istio-proxy/0.log" Oct 30 10:03:08.198037 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.198003 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mj9w5_c9e051aa-16cd-43b2-aec2-4f19f35ab485/kube-rbac-proxy/0.log" Oct 30 10:03:08.321147 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.321110 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6qzz_61e96f32-b0b7-40b7-97e5-61ff980cc465/dns-node-resolver/0.log" Oct 30 10:03:08.938204 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.938165 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 10:03:08.938782 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:08.938745 4920 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" podUID="2ccda3a0-e93c-4d5b-a694-51e310cab997" containerName="discovery" containerID="cri-o://40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb" gracePeriod=30 Oct 30 10:03:09.208623 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.208591 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97vfn_2ccda3a0-e93c-4d5b-a694-51e310cab997/discovery/0.log" Oct 30 10:03:09.225564 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.225530 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 10:03:09.393810 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.393775 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.393810 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.393819 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.394110 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.393852 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.394110 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.393909 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.394110 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.393969 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.394601 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.394019 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.394836 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.394812 4920 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\") pod \"2ccda3a0-e93c-4d5b-a694-51e310cab997\" (UID: \"2ccda3a0-e93c-4d5b-a694-51e310cab997\") " Oct 30 10:03:09.395297 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.394483 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:03:09.396792 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.396741 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:03:09.396792 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.396753 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:03:09.396941 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.396926 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "kube-api-access-chb62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:03:09.396989 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.396954 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:03:09.397300 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.397276 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:03:09.401449 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.401416 4920 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs" pod "2ccda3a0-e93c-4d5b-a694-51e310cab997" (UID: "2ccda3a0-e93c-4d5b-a694-51e310cab997"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:03:09.420938 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.420897 4920 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64jq8_0a50c02c-9976-4034-8785-68c3e971060a/konnectivity-agent/0.log" limit=1024 Oct 30 10:03:09.495929 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495836 4920 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-dns-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.495929 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495874 4920 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chb62\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-kube-api-access-chb62\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.495929 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495890 4920 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-kubeconfig\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.495929 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495904 4920 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ccda3a0-e93c-4d5b-a694-51e310cab997-cacerts\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.495929 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495921 4920 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.496296 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495934 4920 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ccda3a0-e93c-4d5b-a694-51e310cab997-istio-csr-ca-configmap\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.496296 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.495947 4920 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccda3a0-e93c-4d5b-a694-51e310cab997-local-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Oct 30 10:03:09.524953 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.524919 4920 generic.go:356] "Generic (PLEG): container finished" podID="2ccda3a0-e93c-4d5b-a694-51e310cab997" containerID="40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb" exitCode=0 Oct 30 10:03:09.524953 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.524960 4920 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" Oct 30 10:03:09.525183 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.524992 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" event={"ID":"2ccda3a0-e93c-4d5b-a694-51e310cab997","Type":"ContainerDied","Data":"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb"} Oct 30 10:03:09.525183 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.525044 4920 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn" event={"ID":"2ccda3a0-e93c-4d5b-a694-51e310cab997","Type":"ContainerDied","Data":"6e40db0a67c244057482d41c1e2ad828eeacd2f4cea71f9fdd41ae26f8ec6ea4"} Oct 30 10:03:09.525183 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.525105 4920 scope.go:122] "RemoveContainer" containerID="40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb" Oct 30 10:03:09.549147 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.549123 4920 scope.go:122] "RemoveContainer" containerID="40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb" Oct 30 10:03:09.549527 ip-10-0-129-105 kubenswrapper[4920]: E1030 10:03:09.549503 4920 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb\": container with ID starting with 40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb not found: ID does not exist" containerID="40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb" Oct 30 10:03:09.549631 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.549540 4920 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb"} err="failed to get container status \"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb\": rpc error: code = NotFound desc = could not find container \"40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb\": container with ID starting with 40304e3b2110f7608ee48fdc4a639534e216fdf4d6f6ae53e8a8c3ed5bea6dbb not found: ID does not exist" Oct 30 10:03:09.576783 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.576742 4920 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 10:03:09.580498 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.580468 4920 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97vfn"] Oct 30 10:03:09.969295 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:09.969268 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zzzz_8d2f4c5d-a352-4c80-8794-edf020ddbafa/node-ca/0.log" Oct 30 10:03:10.398104 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:10.398048 4920 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ccda3a0-e93c-4d5b-a694-51e310cab997" path="/var/lib/kubelet/pods/2ccda3a0-e93c-4d5b-a694-51e310cab997/volumes" Oct 30 10:03:11.985490 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:11.985456 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97vfn_2ccda3a0-e93c-4d5b-a694-51e310cab997/discovery/0.log" Oct 30 10:03:11.989651 ip-10-0-129-105 kubenswrapper[4920]: I1030 10:03:11.989624 4920 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97vfn_2ccda3a0-e93c-4d5b-a694-51e310cab997/discovery/0.log"